Bug 1839414 - Login to Gnome with AD unqualified login name fails for some users after upgrade to F32
Summary: Login to Gnome with AD unqualified login name fails for some users after upgr...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: sssd
Version: 32
Hardware: x86_64
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Michal Zidek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-24 00:29 UTC by René Genz
Modified: 2020-06-08 14:56 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-08 14:56:35 UTC
Type: Bug


Attachments (Terms of Use)
systemctl stop sssd ; /usr/sbin/sssd -i -d9 (1.58 MB, text/plain)
2020-05-24 00:29 UTC, René Genz
no flags Details

Description René Genz 2020-05-24 00:29:38 UTC
Created attachment 1691397 [details]
systemctl stop sssd ; /usr/sbin/sssd -i -d9

Description of problem:
We manage user accounts in Microsoft Active Directory (AD). Some accounts have POSIX attributes set in AD, some don't. Domain controllers run on "Microsoft Windows Server 2012 R2".
After upgrading from Fedora 30 to Fedora 32 some users cannot log in with GDM (SSH fails too, but users do not use it). So far I have seen only problems with user accounts without POSIX attributes set in AD.
I compared the affected user accounts and their settings to no avail. I cannot find a pattern. I cannot find out why only some users are affected. On the same computer "userw" can log in, but "userf" cannot log in. The correct password is entered.


No login problem with Fedora 30 or 31. Upgrading to Fedora 32 denies access for "userf". Reinstalling the computer with Fedora 30 allows userf to log in again.
The computers are installed unattended (all have same configuration) and are joined to Active Directory with `realmd` and minor modifications to sssd.conf and krb5.conf. We use Xorg, instead of Wayland, as far as I can see it does not make a difference.
In the following text and attachment I replaced the actual domain name with DOMAIN or domain, depending on capitalization. Same for first name, last name, and login name. I removed timestamps from most of the text to decrease file size.


Retrieving user information works:
root# getent passwd userf
userf:*:234002575:234000513:FIRSTF LASTF:/home/userf@domain.local:/bin/bash
root# getent passwd userw
userw:*:234015489:234000513:FIRSTW LASTW:/home/userw@domain.local:/bin/bash


Version-Release number of selected component (if applicable):
# rpm -qa | grep sssd
sssd-2.2.3-13.fc32.x86_64
sssd-common-pac-2.2.3-13.fc32.x86_64
sssd-krb5-2.2.3-13.fc32.x86_64
sssd-kcm-2.2.3-13.fc32.x86_64
sssd-ipa-2.2.3-13.fc32.x86_64
sssd-nfs-idmap-2.2.3-13.fc32.x86_64
sssd-proxy-2.2.3-13.fc32.x86_64
sssd-client-2.2.3-13.fc32.x86_64
sssd-ldap-2.2.3-13.fc32.x86_64
sssd-krb5-common-2.2.3-13.fc32.x86_64
sssd-common-2.2.3-13.fc32.x86_64
sssd-ad-2.2.3-13.fc32.x86_64

# rpm -qa | grep ^systemd
systemd-rpm-macros-245.4-1.fc32.noarch
systemd-pam-245.4-1.fc32.x86_64
systemd-245.4-1.fc32.x86_64
systemd-libs-245.4-1.fc32.x86_64
systemd-udev-245.4-1.fc32.x86_64

# rpm -q krb5-libs
krb5-libs-1.18-1.fc32.x86_64


How reproducible:
100%

Steps to Reproduce:
1. upgrade from F30 to F32 with `dnf system-upgrade download -y --releasever=32 --allowerasing`
2. Try to log in to Gnome

Actual results:
Login fails. Text below password input box: "Sorry, that didn't work. Please try again."

Expected results:
Login works

Additional info:
We use localization. It did not make a difference in my tests.
root# localectl
   System Locale: LANG=de_DE.UTF-8
       VC Keymap: de
      X11 Layout: de
       X11 Model: pc105
     X11 Options: terminate:ctrl_alt_bksp



### GDM login attempt of userf
root@ittc-rg-1# journalctl -f
...
[sssd[krb5_child[2560]]][2560]: KDC has no support for encryption type
audit[2553]: USER_AUTH pid=2553 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:xdm_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct="userf" exe="/usr/libexec/gdm-session-worker" hostname=ittc-rg-1.domain.local addr=? terminal=/dev/tty1 res=failed'
[sssd[krb5_child[2560]]][2560]: KDC has no support for encryption type
gdm-password][2553]: pam_sss(gdm-password:auth): authentication failure; logname= uid=0 euid=0 tty=/dev/tty1 ruser= rhost= user=userf
[sssd[krb5_child[2560]]][2560]: KDC has no support for encryption type
gdm-password][2553]: pam_sss(gdm-password:auth): received for user userf: 4 (System error)
gdm-password][2553]: gkr-pam: unable to locate daemon control file
gdm-password][2553]: gkr-pam: stashed password to try later in open session
audit[2553]: USER_LOGIN pid=2553 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:xdm_t:s0-s0:c0.c1023 msg='uid=234002575 exe="/usr/libexec/gdm-session-worker" hostname=? addr=? terminal=? res=failed'
ittc-rg-1.domain.local systemd[1]: Stopping User Manager for UID 234015489...
...



### SSH login attempt of userf
root@ittc-rg-1# journalctl -f
...
systemd[1]: Started SSSD Kerberos Cache Manager.
sssd[kcm][2618]: Starting up
[sssd[krb5_child[2619]]][2619]: KDC has no support for encryption type
audit[2615]: USER_AUTH pid=2615 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct="userf" exe="/usr/sbin/sshd" hostname=::1 addr=::1 terminal=ssh res=failed'
[sssd[krb5_child[2619]]][2619]: KDC has no support for encryption type
sshd[2615]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1 user=userf
[sssd[krb5_child[2619]]][2619]: KDC has no support for encryption type
sshd[2615]: pam_sss(sshd:auth): received for user userf: 4 (System error)
sshd[2615]: Failed password for invalid user userf from ::1 port 44220 ssh2
...



root# ls -l /etc/sssd/sssd.conf
-rw-------. 1 root root 487 May 24 00:45 /etc/sssd/sssd.conf

root# cat /etc/sssd/sssd.conf
[domain/domain.local]
override_homedir = /home/%u@%d
override_shell = /bin/bash
cache_credentials = True
ad_site = domain.local
default_shell = /bin/bash
krb5_store_password_if_offline = True
krb5_realm = DOMAIN.LOCAL
realmd_tags = manages-system joined-with-adcli 
id_provider = ad
fallback_homedir = /home/%u@%d
ad_domain = domain.local
use_fully_qualified_names = False
ldap_id_mapping = True
access_provider = ad

[sssd]
domains = domain.local
config_file_version = 2
services = nss, pam



root# cat /etc/krb5.conf
# To opt out of the system crypto-policies configuration of krb5, remove the
# symlink at /etc/krb5.conf.d/crypto-policies which will not be recreated.
includedir /etc/krb5.conf.d/

[logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

[libdefaults]
    dns_canonicalize_hostname = false
    dns_lookup_realm = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    rdns = false
    pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
    spake_preauth_groups = edwards25519
#    default_realm = EXAMPLE.COM
    default_ccache_name = KEYRING:persistent:%{uid}

[realms]
# EXAMPLE.COM = {
#     kdc = kerberos.example.com
#     admin_server = kerberos.example.com
# }

[domain_realm]
# .example.com = EXAMPLE.COM
# example.com = EXAMPLE.COM

Comment 1 Alexander Bokovoy 2020-05-24 04:54:52 UTC
This looks like a duplicate of Bug 1837749.
Can you try installing latest systems version from rawhide to confirm it?

Comment 2 Alexander Bokovoy 2020-05-24 04:55:48 UTC
(In reply to Alexander Bokovoy from comment #1)
> This looks like a duplicate of Bug 1837749.
> Can you try installing latest systems version from rawhide to confirm it?

I meant systemd.

Comment 3 Alexander Bokovoy 2020-05-24 04:59:13 UTC
Ah sorry, this is rc4 removal issue, not systemd.

MIT Kerberos in f32 removed rc4 use. You need to set group policy to force use of AES encryption types in AD side.

Comment 4 Alexander Bokovoy 2020-05-24 05:02:44 UTC
Here's relevant Fedora change: https://fedoraproject.org/wiki/Changes/krb5_crypto_modernization

Comment 5 René Genz 2020-05-24 22:05:17 UTC
Thank you very much for your help! At first the things that did not help to solve the problem:
In regards to duplicate of Bug 1837749 :
I forgot to write in my initial post that I tried upgrading to Rawhide and encountered the same problem. For good measure I downloaded from the website:
https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/s/
the packages:
systemd-245.5-2.fc33.x86_64
systemd-libs-245.5-2.fc33.x86_64
systemd-pam-245.5-2.fc33.x86_64
systemd-rpm-macros-245.5-2.fc33.noarch
systemd-udev-245.5-2.fc33.x86_64
and installed them with:
root# dnf install -y ./*.rpm ; reboot

The log in problem persists - userf cannot log in (same error message), userw can log in.


In regards to the wiki article:
In "/etc/krb5.conf" in "[libdefaults]" setting either of these did not work for me on F32:
allow_weak_rc4 = true
allow_weak_crypto = rc4



Now the thing that help to solve the problem:
I can confirm that in "/etc/krb5.conf.d/crypto-policies" appending "arcfour-hmac-md5" to "permitted_enctypes" and restarting the F32 computer does allow log in for userf! :)
`update-crypto-policies --set LEGACY && reboot` works too, but changes other crypto settings too, hence reverted with `update-crypto-policies --set DEFAULT && reboot`.



Now a question and more information:
I assumed all users would have been affected. Why are only some users affected?
So far I could not find data in Active Directory that allows me to predict in advance which user account will have problems and which one will not. The affected user accounts were created in the year 2003, 2008, and 2019. One of them changed the password in Active Directory last week to no avail. The un- and the affected user accounts are in the same OU. All users started with Fedora 30 this March. In the past they used Microsoft Windows 7 (and before that Windows XP; we started with Windows Server 2003 and kept on upgrading as far as I know).
On Fedora 30 userw and userf get the same output (when you ignore the timestamps that are different by a few minutes):
$ klist -ef
Ticketzwischenspeicher: KCM:234015489:68868
Standard-Principal: userw@DOMAIN.LOCAL

Valid starting       Expires              Service principal
24.05.2020 20:37:39  25.05.2020 06:37:39  krbtgt/DOMAIN.LOCAL@DOMAIN.LOCAL
	erneuern bis 31.05.2020 20:37:39, Schalter: FRIA
	Etype (Skey, TKT): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 


On Fedora 32 (with the above workaround for userf) it is the same output.

In dsa.msc in the "Domain controller" OU the domain controllers' msDS-SupportedEncryptionTypes attribute is set to:
RC4_HMAC_MD5 | AES128_CTS_HMAC_SHA1_96 | AES256_CTS_HMAC_SHA1_96
The functional domain level is "Windows Server 2012 R2" too.
For userf and userw the msDS-SupportedEncryptionTypes attribute is not set.


I will ask my colleagues to set a group policy in AD forcing use of AES encryption types. I will post results.

Comment 6 René Genz 2020-06-08 14:56:35 UTC
For our environment we did not had to write a GPO. Only RC4, AES128, and AES256 keys are allowed in Active Directory (AD), see comment 5. New user accounts use AES keys by default. The Fedora 32 client takes RC4 out of the equation. We found out that:
- one user mistyped the password
- the other users had only old keys stored in AD [1]. We were able to verify on Microsoft Windows 10 [2] and Fedora 32 [3].

To resolve the issue affected users reset their password, we waited 30 minutes to complete the synchronization between the domain controllers. After that users had AES keys stored in AD and were able to log in to Fedora 32.


Side note: We noticed that like 1 out of 50 (or even more) computers that join automatically with preset computer objects, `realm join` prints the same error message "KDC has no support for encryption type".
The computer objects are preset with `adcli` on CentOS 7 server. AD consists of 2 domain controllers in master-master mode.
We continue investigation. Maybe this is a local problem too. If we find something useful, a separate message will follow.



[1]
The information "which keys are stored for object X" is not easily available. "dsa.msc -- Attribute Editor" does not provide it. The AD database and the global AD-encryption key had to be exported by Volume Shadow Copy Service (VSS). Afterwards the file ntds.dit had to be searched.

* open PowerShell with administrative rights ; you might need to install a PowerShell extension
* commands ("> " at the start of the line is the prompt):
> $key = Get-BootKey -SystemHivePath 'd:ad\System'
> Get-ADDBAccount -SamAccountName USERNAME -DatabasePath 'd:\ad\ntds.dit' -BootKey $key > c:\temp\USERNAME.txt

* in the output you will see near the end for an account without AES keys:
...
    Kerberos:
      Credentials:
        DES_CBC_MD5
          Key: ...
        DES_CBC_CRC
          Key: ...
      Old Credentials:
        ... # repeating DES_CBC_MD5 and DES_CBC_CRC
      Salt: REALMUSERNAME
      Flags: 0
    ...


For an account with AES keys:
...
    Kerberos:
      Credentials:
        AES256_CTS_HMAC_SHA1_96
          Key: ...
          Iterations: ...
        AES128_CTS_HMAC_SHA1_96
          Key: ...
          Iterations: ...
        DES_CBC_MD5
          Key: ...
          Iterations: ...
      OldCredentials:
        ... # similar to above
      OlderCredentials:
        ... # similar to above
      ServiceCredentials:
      Salt: REALMUSERNAME
      DefaultIterationCount: ...
      Flags: 0
    ...



[2]
Testing Microsoft Windows:
* install "AdoptOpenJDK" (provides Kerberos tools)
* open cmd.exe
* commands ("> " at the start of the line is the prompt):

For an account without AES keys:
> kinit USERNAME dummypassword
Exception: krb_error 14 KDC has no support for encryption type


For an account with AES keys, but wrong password:
> kinit USERNAME dummypassword
Exception: krb_error 24 Pre-authentication failed ...
KrbException: Pre-authentication in...


For an account with AES keys and correct password:
> kinit USERNAME
Password for USERNAME@REALM:
New ticket is stored in cache file C:\Users\USERNAME\krb5cc_USERNAME

> klist -e

Credentials cache: C:\Users\USERNAME\krb5cc_USERNAME

[1]	Service Principal:	krbtgt/REALM@REALM
	Valid starting:		<date and time>
	Expires:		<date and time>
	EType (skey, tkt):	AES256 CTS mode with HMAC SHA1-96, AES256 CTS mode with HMAC SHA1-96



[3]
For testing on Fedora 32 install tools: $ sudo dnf install -y krb5-workstation
REALM must always be uppercase :)

For an account without AES keys (note: no password prompt):
root@... # kinit USERNAME@REALM
kinit: KDC unterstützt diesen Verschlüsselungstyp nicht bei Anfängliche Anmeldedaten werden geholt.


For an account with AES keys and correct password:
root@... # kinit USERNAME@REALM
Password for USERNAME@REALM:


Note You need to log in before you can comment on or make changes to this bug.