Created attachment 1691397 [details] systemctl stop sssd ; /usr/sbin/sssd -i -d9 Description of problem: We manage user accounts in Microsoft Active Directory (AD). Some accounts have POSIX attributes set in AD, some don't. Domain controllers run on "Microsoft Windows Server 2012 R2". After upgrading from Fedora 30 to Fedora 32 some users cannot log in with GDM (SSH fails too, but users do not use it). So far I have seen only problems with user accounts without POSIX attributes set in AD. I compared the affected user accounts and their settings to no avail. I cannot find a pattern. I cannot find out why only some users are affected. On the same computer "userw" can log in, but "userf" cannot log in. The correct password is entered. No login problem with Fedora 30 or 31. Upgrading to Fedora 32 denies access for "userf". Reinstalling the computer with Fedora 30 allows userf to log in again. The computers are installed unattended (all have same configuration) and are joined to Active Directory with `realmd` and minor modifications to sssd.conf and krb5.conf. We use Xorg, instead of Wayland, as far as I can see it does not make a difference. In the following text and attachment I replaced the actual domain name with DOMAIN or domain, depending on capitalization. Same for first name, last name, and login name. I removed timestamps from most of the text to decrease file size. Retrieving user information works: root# getent passwd userf userf:*:234002575:234000513:FIRSTF LASTF:/home/userf:/bin/bash root# getent passwd userw userw:*:234015489:234000513:FIRSTW LASTW:/home/userw:/bin/bash Version-Release number of selected component (if applicable): # rpm -qa | grep sssd sssd-2.2.3-13.fc32.x86_64 sssd-common-pac-2.2.3-13.fc32.x86_64 sssd-krb5-2.2.3-13.fc32.x86_64 sssd-kcm-2.2.3-13.fc32.x86_64 sssd-ipa-2.2.3-13.fc32.x86_64 sssd-nfs-idmap-2.2.3-13.fc32.x86_64 sssd-proxy-2.2.3-13.fc32.x86_64 sssd-client-2.2.3-13.fc32.x86_64 sssd-ldap-2.2.3-13.fc32.x86_64 sssd-krb5-common-2.2.3-13.fc32.x86_64 sssd-common-2.2.3-13.fc32.x86_64 sssd-ad-2.2.3-13.fc32.x86_64 # rpm -qa | grep ^systemd systemd-rpm-macros-245.4-1.fc32.noarch systemd-pam-245.4-1.fc32.x86_64 systemd-245.4-1.fc32.x86_64 systemd-libs-245.4-1.fc32.x86_64 systemd-udev-245.4-1.fc32.x86_64 # rpm -q krb5-libs krb5-libs-1.18-1.fc32.x86_64 How reproducible: 100% Steps to Reproduce: 1. upgrade from F30 to F32 with `dnf system-upgrade download -y --releasever=32 --allowerasing` 2. Try to log in to Gnome Actual results: Login fails. Text below password input box: "Sorry, that didn't work. Please try again." Expected results: Login works Additional info: We use localization. It did not make a difference in my tests. root# localectl System Locale: LANG=de_DE.UTF-8 VC Keymap: de X11 Layout: de X11 Model: pc105 X11 Options: terminate:ctrl_alt_bksp ### GDM login attempt of userf root@ittc-rg-1# journalctl -f ... [sssd[krb5_child[2560]]][2560]: KDC has no support for encryption type audit[2553]: USER_AUTH pid=2553 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:xdm_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct="userf" exe="/usr/libexec/gdm-session-worker" hostname=ittc-rg-1.domain.local addr=? terminal=/dev/tty1 res=failed' [sssd[krb5_child[2560]]][2560]: KDC has no support for encryption type gdm-password][2553]: pam_sss(gdm-password:auth): authentication failure; logname= uid=0 euid=0 tty=/dev/tty1 ruser= rhost= user=userf [sssd[krb5_child[2560]]][2560]: KDC has no support for encryption type gdm-password][2553]: pam_sss(gdm-password:auth): received for user userf: 4 (System error) gdm-password][2553]: gkr-pam: unable to locate daemon control file gdm-password][2553]: gkr-pam: stashed password to try later in open session audit[2553]: USER_LOGIN pid=2553 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:xdm_t:s0-s0:c0.c1023 msg='uid=234002575 exe="/usr/libexec/gdm-session-worker" hostname=? addr=? terminal=? res=failed' ittc-rg-1.domain.local systemd[1]: Stopping User Manager for UID 234015489... ... ### SSH login attempt of userf root@ittc-rg-1# journalctl -f ... systemd[1]: Started SSSD Kerberos Cache Manager. sssd[kcm][2618]: Starting up [sssd[krb5_child[2619]]][2619]: KDC has no support for encryption type audit[2615]: USER_AUTH pid=2615 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct="userf" exe="/usr/sbin/sshd" hostname=::1 addr=::1 terminal=ssh res=failed' [sssd[krb5_child[2619]]][2619]: KDC has no support for encryption type sshd[2615]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1 user=userf [sssd[krb5_child[2619]]][2619]: KDC has no support for encryption type sshd[2615]: pam_sss(sshd:auth): received for user userf: 4 (System error) sshd[2615]: Failed password for invalid user userf from ::1 port 44220 ssh2 ... root# ls -l /etc/sssd/sssd.conf -rw-------. 1 root root 487 May 24 00:45 /etc/sssd/sssd.conf root# cat /etc/sssd/sssd.conf [domain/domain.local] override_homedir = /home/%u@%d override_shell = /bin/bash cache_credentials = True ad_site = domain.local default_shell = /bin/bash krb5_store_password_if_offline = True krb5_realm = DOMAIN.LOCAL realmd_tags = manages-system joined-with-adcli id_provider = ad fallback_homedir = /home/%u@%d ad_domain = domain.local use_fully_qualified_names = False ldap_id_mapping = True access_provider = ad [sssd] domains = domain.local config_file_version = 2 services = nss, pam root# cat /etc/krb5.conf # To opt out of the system crypto-policies configuration of krb5, remove the # symlink at /etc/krb5.conf.d/crypto-policies which will not be recreated. includedir /etc/krb5.conf.d/ [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [libdefaults] dns_canonicalize_hostname = false dns_lookup_realm = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true rdns = false pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt spake_preauth_groups = edwards25519 # default_realm = EXAMPLE.COM default_ccache_name = KEYRING:persistent:%{uid} [realms] # EXAMPLE.COM = { # kdc = kerberos.example.com # admin_server = kerberos.example.com # } [domain_realm] # .example.com = EXAMPLE.COM # example.com = EXAMPLE.COM
This looks like a duplicate of Bug 1837749. Can you try installing latest systems version from rawhide to confirm it?
(In reply to Alexander Bokovoy from comment #1) > This looks like a duplicate of Bug 1837749. > Can you try installing latest systems version from rawhide to confirm it? I meant systemd.
Ah sorry, this is rc4 removal issue, not systemd. MIT Kerberos in f32 removed rc4 use. You need to set group policy to force use of AES encryption types in AD side.
Here's relevant Fedora change: https://fedoraproject.org/wiki/Changes/krb5_crypto_modernization
Thank you very much for your help! At first the things that did not help to solve the problem: In regards to duplicate of Bug 1837749 : I forgot to write in my initial post that I tried upgrading to Rawhide and encountered the same problem. For good measure I downloaded from the website: https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/s/ the packages: systemd-245.5-2.fc33.x86_64 systemd-libs-245.5-2.fc33.x86_64 systemd-pam-245.5-2.fc33.x86_64 systemd-rpm-macros-245.5-2.fc33.noarch systemd-udev-245.5-2.fc33.x86_64 and installed them with: root# dnf install -y ./*.rpm ; reboot The log in problem persists - userf cannot log in (same error message), userw can log in. In regards to the wiki article: In "/etc/krb5.conf" in "[libdefaults]" setting either of these did not work for me on F32: allow_weak_rc4 = true allow_weak_crypto = rc4 Now the thing that help to solve the problem: I can confirm that in "/etc/krb5.conf.d/crypto-policies" appending "arcfour-hmac-md5" to "permitted_enctypes" and restarting the F32 computer does allow log in for userf! :) `update-crypto-policies --set LEGACY && reboot` works too, but changes other crypto settings too, hence reverted with `update-crypto-policies --set DEFAULT && reboot`. Now a question and more information: I assumed all users would have been affected. Why are only some users affected? So far I could not find data in Active Directory that allows me to predict in advance which user account will have problems and which one will not. The affected user accounts were created in the year 2003, 2008, and 2019. One of them changed the password in Active Directory last week to no avail. The un- and the affected user accounts are in the same OU. All users started with Fedora 30 this March. In the past they used Microsoft Windows 7 (and before that Windows XP; we started with Windows Server 2003 and kept on upgrading as far as I know). On Fedora 30 userw and userf get the same output (when you ignore the timestamps that are different by a few minutes): $ klist -ef Ticketzwischenspeicher: KCM:234015489:68868 Standard-Principal: userw Valid starting Expires Service principal 24.05.2020 20:37:39 25.05.2020 06:37:39 krbtgt/DOMAIN.LOCAL erneuern bis 31.05.2020 20:37:39, Schalter: FRIA Etype (Skey, TKT): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 On Fedora 32 (with the above workaround for userf) it is the same output. In dsa.msc in the "Domain controller" OU the domain controllers' msDS-SupportedEncryptionTypes attribute is set to: RC4_HMAC_MD5 | AES128_CTS_HMAC_SHA1_96 | AES256_CTS_HMAC_SHA1_96 The functional domain level is "Windows Server 2012 R2" too. For userf and userw the msDS-SupportedEncryptionTypes attribute is not set. I will ask my colleagues to set a group policy in AD forcing use of AES encryption types. I will post results.
For our environment we did not had to write a GPO. Only RC4, AES128, and AES256 keys are allowed in Active Directory (AD), see comment 5. New user accounts use AES keys by default. The Fedora 32 client takes RC4 out of the equation. We found out that: - one user mistyped the password - the other users had only old keys stored in AD [1]. We were able to verify on Microsoft Windows 10 [2] and Fedora 32 [3]. To resolve the issue affected users reset their password, we waited 30 minutes to complete the synchronization between the domain controllers. After that users had AES keys stored in AD and were able to log in to Fedora 32. Side note: We noticed that like 1 out of 50 (or even more) computers that join automatically with preset computer objects, `realm join` prints the same error message "KDC has no support for encryption type". The computer objects are preset with `adcli` on CentOS 7 server. AD consists of 2 domain controllers in master-master mode. We continue investigation. Maybe this is a local problem too. If we find something useful, a separate message will follow. [1] The information "which keys are stored for object X" is not easily available. "dsa.msc -- Attribute Editor" does not provide it. The AD database and the global AD-encryption key had to be exported by Volume Shadow Copy Service (VSS). Afterwards the file ntds.dit had to be searched. * open PowerShell with administrative rights ; you might need to install a PowerShell extension * commands ("> " at the start of the line is the prompt): > $key = Get-BootKey -SystemHivePath 'd:ad\System' > Get-ADDBAccount -SamAccountName USERNAME -DatabasePath 'd:\ad\ntds.dit' -BootKey $key > c:\temp\USERNAME.txt * in the output you will see near the end for an account without AES keys: ... Kerberos: Credentials: DES_CBC_MD5 Key: ... DES_CBC_CRC Key: ... Old Credentials: ... # repeating DES_CBC_MD5 and DES_CBC_CRC Salt: REALMUSERNAME Flags: 0 ... For an account with AES keys: ... Kerberos: Credentials: AES256_CTS_HMAC_SHA1_96 Key: ... Iterations: ... AES128_CTS_HMAC_SHA1_96 Key: ... Iterations: ... DES_CBC_MD5 Key: ... Iterations: ... OldCredentials: ... # similar to above OlderCredentials: ... # similar to above ServiceCredentials: Salt: REALMUSERNAME DefaultIterationCount: ... Flags: 0 ... [2] Testing Microsoft Windows: * install "AdoptOpenJDK" (provides Kerberos tools) * open cmd.exe * commands ("> " at the start of the line is the prompt): For an account without AES keys: > kinit USERNAME dummypassword Exception: krb_error 14 KDC has no support for encryption type For an account with AES keys, but wrong password: > kinit USERNAME dummypassword Exception: krb_error 24 Pre-authentication failed ... KrbException: Pre-authentication in... For an account with AES keys and correct password: > kinit USERNAME Password for USERNAME@REALM: New ticket is stored in cache file C:\Users\USERNAME\krb5cc_USERNAME > klist -e Credentials cache: C:\Users\USERNAME\krb5cc_USERNAME [1] Service Principal: krbtgt/REALM@REALM Valid starting: <date and time> Expires: <date and time> EType (skey, tkt): AES256 CTS mode with HMAC SHA1-96, AES256 CTS mode with HMAC SHA1-96 [3] For testing on Fedora 32 install tools: $ sudo dnf install -y krb5-workstation REALM must always be uppercase :) For an account without AES keys (note: no password prompt): root@... # kinit USERNAME@REALM kinit: KDC unterstützt diesen Verschlüsselungstyp nicht bei Anfängliche Anmeldedaten werden geholt. For an account with AES keys and correct password: root@... # kinit USERNAME@REALM Password for USERNAME@REALM: