Bug 1839805 - 2.3.0 regression: Can't authenticate against AD any more
Summary: 2.3.0 regression: Can't authenticate against AD any more
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: sssd
Version: 34
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Sumit Bose
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1847847
TreeView+ depends on / blocked
 
Reported: 2020-05-25 15:36 UTC by Martin Pitt
Modified: 2022-03-25 10:46 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1847847 (view as bug list)
Environment:
Last Closed: 2021-12-17 09:09:03 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
sssd debug logs (23.14 KB, application/gzip)
2020-05-25 18:00 UTC, Martin Pitt
no flags Details
sssd debug logs with sssd 2.2.3 on F32 (works) (48.77 KB, application/gzip)
2020-05-26 05:49 UTC, Martin Pitt
no flags Details

Description Martin Pitt 2020-05-25 15:36:38 UTC
Description of problem: Yesterday's sssd upgrade [1] regressed authentication against AD. This was spotted by our fedora-testing image refresh in Cockpit [2], unfortunately it came a day too late to block the bodhi update.

Version-Release number of selected component (if applicable):

sssd-client-2.3.0-1.fc31.x86_64
krb5-libs-1.17-46.fc31.x86_64
adcli-0.8.2-7.fc31.x86_64

How reproducible: Always


Steps to Reproduce:
1. Join an AD domain (we test against Samba):

# realm discover
cockpit.lan
  type: kerberos
  realm-name: COCKPIT.LAN
  domain-name: cockpit.lan
  configured: no
  server-software: active-directory
  client-software: sssd
  required-package: oddjob
  required-package: oddjob-mkhomedir
  required-package: sssd
  required-package: adcli
  required-package: samba-common-tools

# realm join 
[... succeeds]

2. Validate domain user resolution:

# id Administrator
uid=766000500(administrator) gid=766000513(domain users) groups=766000513(domain users),766000519(enterprise admins),766000520(group policy creator owners),766000518(schema admins),766000572(denied rodc password replication group),766000512(domain admins)

3. Try to use it for authentication:

# ssh -l Administrator localhost
Administrator@localhost's password: 
Connection closed by ::1 port 22


Journal output during ssh:

Mai 25 11:21:30 x0.cockpit.lan sshd[9242]: pam_sss(sshd:auth): authentication success; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1 user=Administrator
Mai 25 11:21:30 x0.cockpit.lan audit[9242]: USER_AUTH pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_succeed_if,pam_succeed_if,pam_sss acct="Administrator" exe="/usr/sbin/sshd" hostname=::1 addr=::1 terminal=ssh res=success'
Mai 25 11:21:30 x0.cockpit.lan kernel: audit: type=1100 audit(1590420090.634:523): pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_succeed_if,pam_succeed_if,pam_sss acct="Administrator" exe="/usr/sbin/sshd" hostname=::1 addr=::1 terminal=ssh res=success'
Mai 25 11:21:31 x0.cockpit.lan sshd[9242]: pam_sss(sshd:account): Access denied for user Administrator: 4 (System error)
Mai 25 11:21:31 x0.cockpit.lan sshd[9242]: Failed password for Administrator from ::1 port 48764 ssh2
Mai 25 11:21:31 x0.cockpit.lan sshd[9242]: fatal: Access denied for user Administrator by PAM account configuration [preauth]
Mai 25 11:21:31 x0.cockpit.lan kernel: audit: type=1101 audit(1590420091.564:524): pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=? acct="Administrator" exe="/usr/sbin/sshd" hostname=::1 addr=::1 terminal=ssh res=failed'
Mai 25 11:21:31 x0.cockpit.lan audit[9242]: USER_ACCT pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=? acct="Administrator" exe="/usr/sbin/sshd" hostname=::1 addr=::1 terminal=ssh res=failed'
Mai 25 11:21:31 x0.cockpit.lan audit[9242]: CRYPTO_KEY_USER pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:fd:9e:7c:fc:58:c0:cf:77:3f:8a:ff:1f:40:68:c0:e1:c5:d1:a9:68:ae:49:a4:65:29:4b:c2:14:84:12:7a:11 direction=? spid=9243 suid=74  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
Mai 25 11:21:31 x0.cockpit.lan kernel: audit: type=2404 audit(1590420091.565:525): pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:fd:9e:7c:fc:58:c0:cf:77:3f:8a:ff:1f:40:68:c0:e1:c5:d1:a9:68:ae:49:a4:65:29:4b:c2:14:84:12:7a:11 direction=? spid=9243 suid=74  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
Mai 25 11:21:31 x0.cockpit.lan audit[9242]: CRYPTO_KEY_USER pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=session fp=? direction=both spid=9243 suid=74 rport=48764 laddr=::1 lport=22  exe="/usr/sbin/sshd" hostname=? addr=::1 terminal=? res=success'
Mai 25 11:21:31 x0.cockpit.lan kernel: audit: type=2404 audit(1590420091.565:526): pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=session fp=? direction=both spid=9243 suid=74 rport=48764 laddr=::1 lport=22  exe="/usr/sbin/sshd" hostname=? addr=::1 terminal=? res=success'
Mai 25 11:21:31 x0.cockpit.lan audit[9242]: CRYPTO_KEY_USER pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:fd:9e:7c:fc:58:c0:cf:77:3f:8a:ff:1f:40:68:c0:e1:c5:d1:a9:68:ae:49:a4:65:29:4b:c2:14:84:12:7a:11 direction=? spid=9242 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
Mai 25 11:21:31 x0.cockpit.lan kernel: audit: type=2404 audit(1590420091.574:527): pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:fd:9e:7c:fc:58:c0:cf:77:3f:8a:ff:1f:40:68:c0:e1:c5:d1:a9:68:ae:49:a4:65:29:4b:c2:14:84:12:7a:11 direction=? spid=9242 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
Mai 25 11:21:31 x0.cockpit.lan kernel: audit: type=1112 audit(1590420091.574:528): pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="Administrator" exe="/usr/sbin/sshd" hostname=? addr=::1 terminal=ssh res=failed'
Mai 25 11:21:31 x0.cockpit.lan audit[9242]: USER_LOGIN pid=9242 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="Administrator" exe="/usr/sbin/sshd" hostname=? addr=::1 terminal=ssh res=failed'

I tried "dnf downgrade sssd-ad", but that's just causing sssd.service to fail (due to a version check in /var). But on last week's image with sssd 2.2.3 this works fine.


[1] https://bodhi.fedoraproject.org/updates/FEDORA-2020-803e9e021c
[2] https://github.com/cockpit-project/bots/pull/893

Comment 1 Alexander Bokovoy 2020-05-25 15:39:19 UTC
Make sure your AD environment is configured to use AES encryption types for users. System-wide crypto policy in Fedora 32 and above disables RC4-HMAC.

Comment 2 Martin Pitt 2020-05-25 15:52:59 UTC
@Alexander: So should this be reassigned to realmd then? I'd expect `realm join` to work. Besides, this is Fedora 31, does the changed crypto policy apply to that as well?

Comment 3 Alexander Bokovoy 2020-05-25 16:05:18 UTC
To say anything concrete, we need to see SSSD logs.

Set 

debug_level = 9

in sssd.conf

for [domain/*], [nss], and [pam] sections, restart SSSD and collect /var/log/sssd directory.

Alternatively, you might want to call 'sssctl debug-level 9' before the test if none of applications in the test restarts sssd, this setting will not survive restart.

To pick up sssd logs, you can also do

sssctl logs-fetch /path/to/file.tar

and then attach the file.

Comment 4 Martin Pitt 2020-05-25 18:00:17 UTC
Created attachment 1691988 [details]
sssd debug logs

Thanks Alexander for the hints -- I keep forgetting that sssd doesn't log to the journal. The debug logs are quite a mouthful, so I'm not sure which useful extracts to provide. I naïvely grepped for

# grep -ri rc4.*hmac /var/log/sssd/
/var/log/sssd/krb5_child.log:(2020-05-25 13:55:27): [krb5_child[9501]] [sss_child_krb5_trace_cb] (0x4000): [9501] 1590429327.118054: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, rc4-hmac, camellia128-cts, camellia256-cts

but that doesn't seem to be very useful.

Comment 5 Martin Pitt 2020-05-25 18:10:01 UTC
> Make sure your AD environment is configured to use AES encryption types for users

Do you happen to have some links for that? A Google search isn't very fruitful, and https://wiki.samba.org/index.php/Samba_Security_Documentation seems to suggest that it's supported already.

I checked the entire /etc in the Samba 4.3.11 container for `grep -ri aes` and `grep -ri rc4`, that is the only hit:

krb5kdc/kdc.conf:        supported_enctypes = aes256-cts:normal arcfour-hmac:normal des3-hmac-sha1:normal des-cbc-crc:normal des:normal des:v4 des:norealm des:onlyrealm des:afs3

Comment 6 Alexander Bokovoy 2020-05-25 18:21:08 UTC
there is a failure in retrieving GPOs.

in PAM log:
(2020-05-25 13:55:27): [pam] [pam_dp_send_req] (0x0100): Sending request with the following data:
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): command: SSS_PAM_ACCT_MGMT
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): domain: cockpit.lan
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): user: Administrator
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): service: sshd
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): tty: ssh
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): ruser: not set
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): rhost: ::1
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): authtok type: 0
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): newauthtok type: 0
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): priv: 1
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): cli_pid: 9499
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): logon name: Administrator
(2020-05-25 13:55:27): [pam] [pam_print_data] (0x0100): flags: 0
(2020-05-25 13:55:27): [pam] [pam_dom_forwarder] (0x0100): pam_dp_send_req returned 0
(2020-05-25 13:55:28): [pam] [sbus_dispatch] (0x4000): Dispatching.
(2020-05-25 13:55:28): [pam] [pam_dp_send_req_done] (0x0200): received: [4 (System error)][cockpit.lan]
(2020-05-25 13:55:28): [pam] [pam_reply] (0x4000): pam_reply initially called with result [4]: System error. this result might be changed during processing
(2020-05-25 13:55:28): [pam] [filter_responses] (0x0100): [pam_response_filter] not available, not fatal.
(2020-05-25 13:55:28): [pam] [pam_reply] (0x0200): blen: 28
(2020-05-25 13:55:28): [pam] [pam_reply] (0x0200): Returning [4]: System error to the client

from domain log:
(2020-05-25 13:55:28): [be[cockpit.lan]] [read_pipe_handler] (0x0400): EOF received, client finished
(2020-05-25 13:55:28): [be[cockpit.lan]] [gpo_cse_done] (0x0020): ad_gpo_parse_gpo_child_response failed: [22][Invalid argument]
(2020-05-25 13:55:28): [be[cockpit.lan]] [ad_gpo_cse_done] (0x0400): gpo_guid: {31B2F340-016D-11D2-945F-00C04FB984F9}
(2020-05-25 13:55:28): [be[cockpit.lan]] [ad_gpo_cse_done] (0x0040): Unable to retrieve policy data: [22](Invalid argument}
(2020-05-25 13:55:28): [be[cockpit.lan]] [ad_gpo_access_done] (0x0040): GPO-based access control failed.

from GPO child log:
(2020-05-25 13:55:27): [gpo_child[9502]] [main] (0x0400): performing smb operations
(2020-05-25 13:55:27): [gpo_child[9502]] [copy_smb_file_to_gpo_cache] (0x0400): smb_uri: smb://f0.cockpit.lan/sysvol/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/GPT.INI
(2020-05-25 13:55:28): [gpo_child[9502]] [copy_smb_file_to_gpo_cache] (0x4000): smb_buflen: 20
(2020-05-25 13:55:28): [gpo_child[9502]] [prepare_gpo_cache] (0x4000): smb_path_with_suffix: /cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/GPT.INI
(2020-05-25 13:55:28): [gpo_child[9502]] [prepare_gpo_cache] (0x0400): Storing GPOs in /var/lib/sss/gpo_cache/cockpit.lan
(2020-05-25 13:55:28): [gpo_child[9502]] [prepare_gpo_cache] (0x0400): Storing GPOs in /var/lib/sss/gpo_cache/cockpit.lan/Policies
(2020-05-25 13:55:28): [gpo_child[9502]] [prepare_gpo_cache] (0x0400): Storing GPOs in /var/lib/sss/gpo_cache/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}
(2020-05-25 13:55:28): [gpo_child[9502]] [unique_filename_destructor] (0x2000): Unlinking [/var/lib/sss/gpo_cache/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/GPT.INIDQTKDJ]
(2020-05-25 13:55:28): [gpo_child[9502]] [unlink_dbg] (0x2000): File already removed: [/var/lib/sss/gpo_cache/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/GPT.INIDQTKDJ]
(2020-05-25 13:55:28): [gpo_child[9502]] [ad_gpo_parse_ini_file] (0x0400): ini_filename:/var/lib/sss/gpo_cache/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/GPT.INI
(2020-05-25 13:55:28): [gpo_child[9502]] [perform_smb_operations] (0x0400): sysvol_gpt_version: 0
(2020-05-25 13:55:28): [gpo_child[9502]] [copy_smb_file_to_gpo_cache] (0x0400): smb_uri: smb://f0.cockpit.lan/sysvol/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/Machine/Microsoft/Windows NT/SecEdit/GptTmpl.inf
(2020-05-25 13:55:28): [gpo_child[9502]] [copy_smb_file_to_gpo_cache] (0x0400): smb_uri: smb://f0.cockpit.lan/sysvol/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/MACHINE/Microsoft/Windows NT/SecEdit/GptTmpl.inf
(2020-05-25 13:55:28): [gpo_child[9502]] [copy_smb_file_to_gpo_cache] (0x0020): smbc_getFunctionOpen failed [2][No such file or directory]
(2020-05-25 13:55:28): [gpo_child[9502]] [perform_smb_operations] (0x0020): copy_smb_file_to_gpo_cache failed [2][No such file or directory]
(2020-05-25 13:55:28): [gpo_child[9502]] [main] (0x0020): perform_smb_operations failed.[2][No such file or directory].
(2020-05-25 13:55:28): [gpo_child[9502]] [main] (0x0020): gpo_child failed!


So, basically, libsmbclient-based utility in SSSD cannot talk to your AD DC over SMB protocol for specific files (smb://f0.cockpit.lan/sysvol/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/MACHINE/Microsoft/Windows NT/SecEdit/GptTmpl.inf) but can do that for others (smb://f0.cockpit.lan/sysvol/cockpit.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/GPT.INI). Note that SSSD tried the path twice, with /Machine/ and /MACHINE/ component replacement. This is expected, SSSD has the following comment in this place:

            /*
             * DCs may use upper case names for the main folder, where GPTs are
             * stored. libsmbclient does not allow us to request case insensitive
             * file name lookups on DCs with case sensitive file systems.
             */


This looks to me like a problem on the server side -- we have GPT.INI in the policy but no machine policy in place, thus no way to retrieve the policy. The corresponding code in SSSD did not change since 2017 aside from replacing debug string prefix to [gpo_child[%pid]].

Comment 7 Alexander Bokovoy 2020-05-25 18:25:24 UTC
(In reply to Martin Pitt from comment #5)
> > Make sure your AD environment is configured to use AES encryption types for users
> 
> Do you happen to have some links for that? A Google search isn't very
> fruitful, and https://wiki.samba.org/index.php/Samba_Security_Documentation
> seems to suggest that it's supported already.
> 
> I checked the entire /etc in the Samba 4.3.11 container for `grep -ri aes`
> and `grep -ri rc4`, that is the only hit:
> 
> krb5kdc/kdc.conf:        supported_enctypes = aes256-cts:normal
> arcfour-hmac:normal des3-hmac-sha1:normal des-cbc-crc:normal des:normal
> des:v4 des:norealm des:onlyrealm des:afs3

No, my comment is unrelated. Samba DC should be able to prefer AES encryption types. I thought you are using Microsoft Windows Server.

Comment 8 Nicolas Pöhlmann 2020-05-26 00:22:50 UTC
We run into this bug also on Fedora 32 in V2.3.0, so it's not only restricted to F31.

For a temporary fix, you can downgrade to V2.2.3 with deleting the sss-DB files and rejoining the machine to AD.

Overall it's harder to debug the exact case on F32 right now, because of the AD 'dot' username problem in systemd V245.4 [see: https://ask.fedoraproject.org/t/gui-login-with-sssd-managed-users-fails/6606]. So for a full functional F32 AD integration every one needs sssd V2.2.3 and systemd V245.5 from rawhide which I think isn't the best circumstance to post any log here. But it's the same PAM error on F32.

Comment 9 Sumit Bose 2020-05-26 05:23:59 UTC
Hi,

given the failure in the GPO lookup the issue is most probably related to the changes added by https://pagure.io/SSSD/sssd/issue/3324.

As a workaround you can set 'ad_gpo_access_control' to 'disabled' or 'permissive' in the [domain/...] section of sssd.conf to skip the GPO based access control.

Martin, would it be possible to attach SSSD logs from an older version as well so that I can compare what is the effect of the changes in your environment.

bye,
Sumit

Comment 10 Martin Pitt 2020-05-26 05:49:49 UTC
Created attachment 1692119 [details]
sssd debug logs with sssd 2.2.3 on F32 (works)

(In reply to Sumit Bose from comment #9)

> As a workaround you can set 'ad_gpo_access_control' to 'disabled' or
> 'permissive' in the [domain/...] section of sssd.conf to skip the GPO based
> access control.

That works very nicely indeed, thanks!

> Martin, would it be possible to attach SSSD logs from an older version as
> well so that I can compare what is the effect of the changes in your
> environment.

Gladly -- attached.

Thanks!

Comment 11 Sumit Bose 2020-05-26 11:08:14 UTC
Hi,

thanks for the logs. In the old version the GPO is considered as 'not applicable' and hence SSSD does not try to download it. I the new version, due to changes from https://pagure.io/SSSD/sssd/issue/3324, it now is considered applicable and while trying to load the related data SSSD fails.

I wonder if you can send me some raw data as well. First the LDAP object:

    kinit -k 'X0$@COCKPIT.LAN'
    ldapsearch -H ldap://f0.cockpit.lan -Y GSSAPI -b 'CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit,DC=lan'

and second the sysvol content:

    smbclient -k '\\f0.cockpit.lan\sysvol'
    smb: \> ls cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\
    smb: \> ls cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\Machine\

I would expect that the Machine directory is empty, if not, please follow the path as long as possible.

Thanks.

bye,
Sumit

Comment 12 Martin Pitt 2020-05-26 11:34:48 UTC
@Sumit: If you prefer, I'm happy to give you ssh access to a running instance that has everything set up. Just send me your public SSH key.

# ldapsearch -H ldap://f0.cockpit.lan -Y GSSAPI -b 'CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit,DC=lan'
SASL/GSSAPI authentication started
SASL username: X0$@COCKPIT.LAN
SASL SSF: 256
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit,DC=lan> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# {31B2F340-016D-11D2-945F-00C04FB984F9}, Policies, System, cockpit.lan
dn: CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit
 ,DC=lan
objectClass: top
objectClass: container
objectClass: groupPolicyContainer
cn: {31B2F340-016D-11D2-945F-00C04FB984F9}
instanceType: 4
whenCreated: 20200526112825.0Z
whenChanged: 20200526112825.0Z
displayName: Default Domain Policy
uSNCreated: 3585
uSNChanged: 3585
showInAdvancedViewOnly: TRUE
name: {31B2F340-016D-11D2-945F-00C04FB984F9}
objectGUID:: fVb/256RAUCk0sih9OPuig==
flags: 0
versionNumber: 0
systemFlags: -1946157056
objectCategory: CN=Group-Policy-Container,CN=Schema,CN=Configuration,DC=cockpi
 t,DC=lan
isCriticalSystemObject: TRUE
gPCFunctionalityVersion: 2
gPCFileSysPath: \\cockpit.lan\sysvol\cockpit.lan\Policies\{31B2F340-016D-11D2-
 945F-00C04FB984F9}
gPCMachineExtensionNames: [{35378EAC-683F-11D2-A89A-00C04FBBCFA2}{53D6AB1B-248
 8-11D1-A28C-00C04FB94F17}][{827D319E-6EAC-11D2-A4EA-00C04F79F83A}{803E14A0-B4
 FB-11D0-A0D0-00A0C90F574B}][{B1BE8D72-6EAC-11D2-A4EA-00C04F79F83A}{53D6AB1B-2
 488-11D1-A28C-00C04FB94F17}]
gPCUserExtensionNames: [{3060E8D0-7020-11D2-842D-00C04FA372D4}{3060E8CE-7020-1
 1D2-842D-00C04FA372D4}][{35378EAC-683F-11D2-A89A-00C04FBBCFA2}{0F6B957E-509E-
 11D1-A7CC-0000F87571E3}]
distinguishedName: CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=Sy
 stem,DC=cockpit,DC=lan

# User, {31B2F340-016D-11D2-945F-00C04FB984F9}, Policies, System, cockpit.lan
dn: CN=User,CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC
 =cockpit,DC=lan
objectClass: top
objectClass: container
cn: User
instanceType: 4
whenCreated: 20200526112825.0Z
whenChanged: 20200526112825.0Z
uSNCreated: 3586
uSNChanged: 3586
showInAdvancedViewOnly: TRUE
name: User
objectGUID:: BOTZGXz8RU6WORiszMkG+A==
systemFlags: -1946157056
objectCategory: CN=Container,CN=Schema,CN=Configuration,DC=cockpit,DC=lan
isCriticalSystemObject: TRUE
distinguishedName: CN=User,CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Polici
 es,CN=System,DC=cockpit,DC=lan

# Machine, {31B2F340-016D-11D2-945F-00C04FB984F9}, Policies, System, cockpit.la
 n
dn: CN=Machine,CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System
 ,DC=cockpit,DC=lan
objectClass: top
objectClass: container
cn: Machine
instanceType: 4
whenCreated: 20200526112825.0Z
whenChanged: 20200526112825.0Z
uSNCreated: 3587
uSNChanged: 3587
showInAdvancedViewOnly: TRUE
name: Machine
objectGUID:: GOx13irBnEKIjwCn6yaNOg==
systemFlags: -1946157056
objectCategory: CN=Container,CN=Schema,CN=Configuration,DC=cockpit,DC=lan
isCriticalSystemObject: TRUE
distinguishedName: CN=Machine,CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Pol
 icies,CN=System,DC=cockpit,DC=lan

# search result
search: 4
result: 0 Success

# numResponses: 4
# numEntries: 3


smb: \> ls cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\
NT_STATUS_OBJECT_NAME_INVALID listing \cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\

smb: \> ls cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\Machine\
NT_STATUS_OBJECT_NAME_INVALID listing \cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\Machine\

That didn't seem expected, so I went upwards instead:

smb: \> ls cockpit.lan\Policies\
NT_STATUS_OBJECT_NAME_INVALID listing \cockpit.lan\Policies\
smb: \> ls cockpit.lan\Policies
  Policies                            D        0  Tue May 26 07:28:26 2020

		10258432 blocks of size 1024. 4190900 blocks available

It seems to work better with a "*" at the end (NB I don't really know what I'm doing here):

smb: \> ls cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\*
  .                                   D        0  Tue May 26 07:28:26 2020
  ..                                  D        0  Tue May 26 07:28:26 2020
  GPT.INI                             N       20  Tue May 26 07:28:26 2020
  MACHINE                             D        0  Tue May 26 07:28:26 2020
  USER                                D        0  Tue May 26 07:28:26 2020

		10258432 blocks of size 1024. 4190968 blocks available
smb: \> ls cockpit.lan\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\Machine\*
  .                                   D        0  Tue May 26 07:28:26 2020
  ..                                  D        0  Tue May 26 07:28:26 2020

		10258432 blocks of size 1024. 4190968 blocks available

So Machine/ dir is indeed empty.

Comment 13 Sumit Bose 2020-05-26 12:53:33 UTC
Hi,

thanks for the offer to access you system directly. If you don't mind I'd prefer to continue the ping-pong here because the steps recorded here might help others to debug GPO related issues as well. But please let me know if you prefer otherwise.

I forgot that the SIDs are not added automatically to the ldapsearch output, can you re-run ldapsearch with

    kinit -k 'X0$@COCKPIT.LAN'
    ldapsearch -H ldap://f0.cockpit.lan -Y GSSAPI -b 'CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit,DC=lan' cn flags gPCFunctionalityVersion gPCFileSysPath gPCFileSysPath nTSecurityDescriptor

Thanks.

bye,
Sumit

Comment 14 Martin Pitt 2020-05-26 13:47:22 UTC
# ldapsearch -H ldap://f0.cockpit.lan -Y GSSAPI -b 'CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit,DC=lan' cn flags gPCFunctionalityVersion gPCFileSysPath gPCFileSysPath nTSecurityDescriptor
SASL/GSSAPI authentication started
SASL username: X0$@COCKPIT.LAN
SASL SSF: 256
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit,DC=lan> with scope subtree
# filter: (objectclass=*)
# requesting: cn flags gPCFunctionalityVersion gPCFileSysPath gPCFileSysPath nTSecurityDescriptor 
#

# {31B2F340-016D-11D2-945F-00C04FB984F9}, Policies, System, cockpit.lan
dn: CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=cockpit
 ,DC=lan
cn: {31B2F340-016D-11D2-945F-00C04FB984F9}
flags: 0
gPCFunctionalityVersion: 2
gPCFileSysPath: \\cockpit.lan\sysvol\cockpit.lan\Policies\{31B2F340-016D-11D2-
 945F-00C04FB984F9}

# User, {31B2F340-016D-11D2-945F-00C04FB984F9}, Policies, System, cockpit.lan
dn: CN=User,CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC
 =cockpit,DC=lan
cn: User

# Machine, {31B2F340-016D-11D2-945F-00C04FB984F9}, Policies, System, cockpit.la
 n
dn: CN=Machine,CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System
 ,DC=cockpit,DC=lan
cn: Machine

# search result
search: 4
result: 0 Success

# numResponses: 4
# numEntries: 3

wrt. remote debugging, that makes sense -- I'm happy either way, I just know from myself that I prefer local/direct debugging.

Thanks!

Comment 15 Andrew Gunnerson 2020-05-27 21:11:20 UTC
*** Bug 1840908 has been marked as a duplicate of this bug. ***

Comment 16 Sumit Bose 2020-05-29 16:47:24 UTC
Hi,

thanks for providing access to a test system.

The important items are that a Samba DC was used here and that at least in the initial configuration of the Samba DC the GPO related files for the default domain policy {31B2F340-016D-11D2-945F-00C04FB984F9} are empty.

I'm currently discussing with Samba developers what would be the best way to solve this. However, since in this setup the GPO are empty and as a result using them for access control will give ot benefit using 'ad_gpo_access_control = disabled' is not only a workaround but makes sense.

I keep the ticket open in case the result of the discussion is that SSSD has to deal with it. In this case we have to add code to detect this case and handle it more gracefully.

bye,
Sumit

Comment 17 Martin Pitt 2020-05-30 05:52:16 UTC
Thanks Sumit for your debugging efforts!

Naively it feels like manually setting ad_gpo_access_control=disabled on the client side is a hack/workaround. We shouldn't have "unbreak my computer" options to think about. Of course we could add that to our test suite, but we want our testing to represent reality, not being arbitrarily tweaked to work.

The empty machines policy on the server side sound like a more reasonable root cause -- either this is a valid config -- and sssd then should ignore an empty policy; or it is not a valid config, then we should fix our server configuration, and we can file an issue against https://github.com/Fmstrat/samba-domain/blob/master/init.sh to do the right thing by default. That samba setup script doesn't deal with any policy stuff, so at least the default config in Samba 4.3.8 seems wrong. Possibly this is fixed in newer versions, I'm happy to give that a try.

I didn't find some good docs yet how to manage these GPOs -- I found https://wiki.samba.org/index.php/Managing_local_groups_on_domain_members_via_GPO_restricted_groups and https://wiki.samba.org/index.php/Implementing_System_Policies_with_Samba but they both talk about interactively editing the policies with a Windows client. That's not an option for our automated image builds, but hopefully it can be done on the CLI somehow. Does anyone happen to have a pointer here?

Or would it help to entirely remove the Machine\ directory, so that no GPOs are applied? 

Thanks!

Comment 18 Sumit Bose 2020-06-02 09:22:48 UTC
(In reply to Martin Pitt from comment #17)
> Thanks Sumit for your debugging efforts!
> 
> Naively it feels like manually setting ad_gpo_access_control=disabled on the
> client side is a hack/workaround. We shouldn't have "unbreak my computer"
> options to think about. Of course we could add that to our test suite, but
> we want our testing to represent reality, not being arbitrarily tweaked to
> work.
> 
> The empty machines policy on the server side sound like a more reasonable
> root cause -- either this is a valid config -- and sssd then should ignore
> an empty policy; or it is not a valid config, then we should fix our server
> configuration, and we can file an issue against

Hi,

since this is the default Samba DC configuration and I would expect the Windows clients do not have an issue with the missing file, SSSD should at least ignore the missing GPO file for the default domain policy in this case. 


> https://github.com/Fmstrat/samba-domain/blob/master/init.sh to do the right
> thing by default. That samba setup script doesn't deal with any policy
> stuff, so at least the default config in Samba 4.3.8 seems wrong. Possibly
> this is fixed in newer versions, I'm happy to give that a try.
> 
> I didn't find some good docs yet how to manage these GPOs -- I found
> https://wiki.samba.org/index.php/
> Managing_local_groups_on_domain_members_via_GPO_restricted_groups and
> https://wiki.samba.org/index.php/Implementing_System_Policies_with_Samba but
> they both talk about interactively editing the policies with a Windows
> client. That's not an option for our automated image builds, but hopefully
> it can be done on the CLI somehow. Does anyone happen to have a pointer here?
> 

Yes, the afaik the currently expected way is to use gpmc on a Windows client to create the GPO file.

David Mulder from S.u.S.E. pointed out that you can create the file Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\MACHINE\Microsoft\Windows NT\SecEdit\GptTmpl.inf
manually with the content:

-------- snip --------
[Unicode]
Unicode=yes
[Version]
signature="$CHICAGO$"
Revision=1
-------- snip --------

which represents just an empty policy file with a valid header. The UUID/GUID 31B2.... is always the same and represents the default domain policy.


> Or would it help to entirely remove the Machine\ directory, so that no GPOs
> are applied?

No, this won't work, you have to modify or remove entries in the Samba DC LDAP tree but I think creating the file as described above would be easier.

HTH

bye,
Sumit

> 
> Thanks!

Comment 19 Martin Pitt 2020-06-02 14:24:59 UTC
Amazing, thanks Sumit! Creating that file with specified  contents works great. That's a nice workaround for the time being, and better than skipping the tests.

Comment 20 Charles Butterfield 2020-06-03 18:02:49 UTC
I'm having the same problem on Fedora-32.  Just upgraded to sssd 2.3.0-1.fc32 and my AD authentication stopped working.  The workaround described above also works for me, namely

/etc/sssd/sssd.conf
[domain WHATEVER]
ad_gpo_access_control=disabled

Comment 21 Sumit Bose 2020-06-04 05:29:51 UTC
(In reply to Charles Butterfield from comment #20)
> I'm having the same problem on Fedora-32.  Just upgraded to sssd
> 2.3.0-1.fc32 and my AD authentication stopped working.  The workaround
> described above also works for me, namely
> 
> /etc/sssd/sssd.conf
> [domain WHATEVER]
> ad_gpo_access_control=disabled

Hi,

I've seen that you commented on https://bugzilla.redhat.com/show_bug.cgi?id=1840908 as well. Are you using Samba as a domain controller or a Windows AD DC. In the Samba case this ticket here is the right on in the other case you most probably see the issue described in https://bugzilla.redhat.com/show_bug.cgi?id=1840908.

bye,
Sumit

Comment 22 Sumit Bose 2020-07-02 14:28:25 UTC
Hi,

Possible solution:
a Samba DC should have 'vendorName: Samba Team (https://www.samba.org)' in the rootDSE by default. This can be used to check if we are talking with a Samba DC and if that's the case a missing default policy (31B2F340-016D-11D2-945F-00C04FB984F9) file can be ignored.

bye,
Sumit

Comment 23 Pavel Březina 2020-07-08 11:53:36 UTC
Why are we downloading a nonexistent policy? Do we have the default policy hard coded somewhere?

Comment 24 Sumit Bose 2020-07-08 13:00:37 UTC
(In reply to Pavel Březina from comment #23)
> Why are we downloading a nonexistent policy? Do we have the default policy
> hard coded somewhere?

Hi,

in the Samba DC default installation the default domain policy (31B2F340-016D-11D2-945F-00C04FB984F9) is defined in LDAP but on the sysvol file-share there is no related file for it. Since Windows clients do not have an issue with this SSSD should not have either.

HTH

bye,
Sumit

Comment 25 Ben Cotton 2020-11-03 16:59:12 UTC
This message is a reminder that Fedora 31 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '31'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 31 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 26 Ben Cotton 2021-02-09 16:11:07 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.

Comment 27 Martin Pitt 2021-12-17 09:09:03 UTC
We have nto seen this any more since September 2020, we just forgot to close the bug. Sorry!

Comment 28 Alexey Tikhonov 2022-03-25 10:46:25 UTC
JFTR:

Pushed PR: https://github.com/SSSD/sssd/pull/6051

* `master`
    * 81d67a592315740d13e48b9a29e29baa8cc35638 - ad: gpo: ignore GPO if SecEdit/GptTmpl.inf is missing

Going to be fixed in sssd-2.7+


Note You need to log in before you can comment on or make changes to this bug.