Bug 2006382
| Summary: | IPA Intermittence fetching groups | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Chino Soliard <csoliard> |
| Component: | sssd | Assignee: | Sumit Bose <sbose> |
| Status: | CLOSED ERRATA | QA Contact: | Jakub Vavra <jvavra> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.9 | CC: | aboscatt, agk, atikhono, grajaiya, jhrozek, jreznik, jvavra, kjavier, lslebodn, mzidek, pasik, pbrezina, rcritten, rick.morrison, sbose, sgadekar, tapazogl, tscherf, vvanhaft |
| Target Milestone: | rc | Keywords: | Triaged, ZStream |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | sync-to-jira | ||
| Fixed In Version: | sssd-1.16.5-10.el7_9.12 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-02-22 17:03:55 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Moving this BZ from IPA to SSSD as proposed by Sumit on https://bugzilla.redhat.com/show_bug.cgi?id=2006382#c18 https://github.com/SSSD/sssd/pull/5883 was merged upstream, but not yet to 1-16 branch. 1-16 backport: https://github.com/SSSD/sssd/pull/5909 Pushed PR: https://github.com/SSSD/sssd/pull/5909 * `sssd-1-16` * be3ee30c68dd9d2e5184da226dfbe66f516a4b92 - cldap: use dns_resolver_server_timeout timeout for cldap ping * ed243335d3e74ab2cde49eacc9a85ca5408a8dec - ad: only send cldap-ping to our local domain * 664ae9d2247b5139d2286975228baa0cea39a8e4 - ad: make ad_srv_plugin_ctx_switch_site() public * f60a6fc682646a8c16fa8875456300c61cf3e979 - ad: use already discovered forest name * ecfb7df52fd3b0edf8549d42cfa6b378407fb982 - ad: move current site and forest name to a more global context * 4b59b0af3d97b2a6b0acc08fa80377a5f59e5bfe - ad: require name when looking up root domain * 4b02c9632c44265124e4fa130170bc86de369b8c - SSSD man: man_dns_resolver_parameter_modification Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (sssd bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0627 |
Description of problem: IPA clients are having intermittent issues login in with AD user. Environment is using HBAC and sometimes groups are not being fetched correctly causing intermittent login issues. Seems that Trust is failing. Note: running 'ipa-adtrust-install' did not fix the issue. Environment is: - 4 IPA servers - AD trust - They are using POSIX attributes on AD groups ------------------------------------------------------------------------ On IPA Servers, we can see that: - SSSD BE is going offline intermittently: - Winbind is failing Sep 20 01:53:02 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab***** Sep 20 01:53:15 hostname smbd[PID]: [2021/09/20 01:53:15.070342, 0] ../../source3/auth/auth_winbind.c:143(check_winbind_security) Sep 20 01:53:15 hostname smbd[PID]: check_winbind_security: winbindd not running - but required as DC with trusts: NT_STATUS_NO_LOGON_SERVERS Sep 20 01:53:42 hostname sssd[sssd]: Child [PID5] ('example.net':'%BE_example.net') was terminated by own WATCHDOG. Consult corresponding logs to figure out the reason. Sep 20 01:53:42 hostname sssd[be[example.net]]: Starting up Sep 20 01:53:42 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab***** Sep 20 01:53:42 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab***** Sep 20 01:53:42 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab***** Sep 20 01:53:42 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab***** Sep 20 01:53:45 hostname smbd[PID2]: [2021/09/20 01:53:45.115741, 0] ../../source3/auth/auth_winbind.c:143(check_winbind_security) Sep 20 01:53:45 hostname smbd[PID2]: check_winbind_security: winbindd not running - but required as DC with trusts: NT_STATUS_NO_LOGON_SERVERS Sep 20 01:54:05 hostname smbd[PID3]: [2021/09/20 01:54:05.125284, 0] ../../source3/auth/auth_winbind.c:143(check_winbind_security) Sep 20 01:54:05 hostname smbd[PID3]: check_winbind_security: winbindd not running - but required as DC with trusts: NT_STATUS_NO_LOGON_SERVERS Sep 20 01:54:25 hostname smbd[PID4]: [2021/09/20 01:54:25.162790, 0] ../../source3/auth/auth_winbind.c:143(check_winbind_security) Sep 20 01:54:25 hostname smbd[PID4]: check_winbind_security: winbindd not running - but required as DC with trusts: NT_STATUS_NO_LOGON_SERVERS Sep 20 01:55:33 hostname sssd[sssd]: Child [PID6] ('example.net':'%BE_example.net') was terminated by own WATCHDOG. Consult corresponding logs to figure out the reason. Sep 20 01:55:33 hostname sssd[be[example.net]]: Starting up Sep 20 01:55:33 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab***** Sep 20 01:55:33 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab***** Note: In this particular moment, both services failed together, but they commonly fail lonely. ------------------------------------------------------------------------ Here's the failure on the IPA Client to look up the AD User: (2021-08-26 8:16:34): [be[example.net]] [ipa_s2n_get_acct_info_send] (0x0400): Sending request_type: [REQ_FULL_WITH_MEMBERS] for trust user [ADuser] to IPA server (2021-08-26 8:16:34): [be[example.net]] [ipa_s2n_exop_send] (0x0400): Executing extended operation (2021-08-26 8:16:34): [be[example.net]] [ipa_s2n_exop_send] (0x2000): ldap_extended_operation sent, msgid = 109 (2021-08-26 8:16:34): [be[example.net]] [sdap_op_add] (0x2000): New operation 109 timeout 6 (2021-08-26 8:16:34): [be[example.net]] [sdap_process_result] (0x2000): Trace: sh[0x5561682f4850], connected[1], ops[0x5561682dd2a0], ldap[0x55616827c120] (2021-08-26 8:16:34): [be[example.net]] [sdap_process_result] (0x2000): Trace: end of ldap_result list (2021-08-26 8:16:36): [be[example.net]] [sdap_process_result] (0x2000): Trace: sh[0x5561682f4850], connected[1], ops[0x5561682dd2a0], ldap[0x55616827c120] (2021-08-26 8:16:36): [be[example.net]] [sdap_process_message] (0x4000): Message type: [LDAP_RES_EXTENDED] (2021-08-26 8:16:36): [be[example.net]] [ipa_s2n_exop_done] (0x0040): ldap_extended_operation result: Time limit exceeded(3), (null). (2021-08-26 8:16:36): [be[example.net]] [ipa_s2n_exop_done] (0x0040): ldap_extended_operation failed, server logs might contain more details. (2021-08-26 8:16:36): [be[example.net]] [sdap_op_destructor] (0x2000): Operation 109 finished (2021-08-26 8:16:36): [be[example.net]] [ipa_s2n_get_user_done] (0x0040): s2n exop request failed. (2021-08-26 8:16:36): [be[example.net]] [sdap_id_op_done] (0x4000): releasing operation connection (2021-08-26 8:16:36): [be[example.net]] [ipa_subdomain_account_done] (0x0040): ipa_get_*_acct request failed: [1432158230]: Network I/O Error. Seems like a trust failure (s2n exop request failed) ------------------------------------------------------------------------ IPA Servers -> Samba debug logs -> log.winbind-idmap: [2021/09/08 08:54:07.763659, 1] ../../source3/winbindd/idmap_tdb_common.c:66(idmap_tdb_common_allocate_id_action) Fatal Error: GID range full!! (max: 0) [2021/09/08 08:54:07.763682, 1] ../../source3/winbindd/idmap_tdb_common.c:138(idmap_tdb_common_allocate_id) Error allocating a new GID ------------------------------------------------------------------------ SAMBA config: [global] create krb5 conf = No dedicated keytab file = /etc/samba/samba.keytab disable spoolss = Yes domain logons = Yes domain master = Yes kerberos method = dedicated keytab ldap group suffix = cn=groups,cn=accounts ldap machine suffix = cn=computers,cn=accounts ldap ssl = no ldap suffix = dc=ipa,dc=sjrb,dc=ad ldap user suffix = cn=users,cn=accounts log file = /var/log/samba/log.%m max log size = 100000 max smbd processes = 1000 passdb backend = ipasam:ldapi://%2fvar%2frun%2fslapd-IPA-SJRB-AD.socket realm = IPA.SJRB.AD registry shares = Yes security = USER workgroup = IPA idmap config ipa : range = 686600000 - 686800000 idmap config ipa : backend = sss idmap config * : range = 0 - 0 rpc_daemon:lsasd = fork rpc_daemon:epmd = fork rpc_server:tcpip = yes rpc_server:netlogon = external rpc_server:samr = external rpc_server:lsasd = external rpc_server:lsass = external rpc_server:lsarpc = external rpc_server:epmapper = external ldapsam:trusted = yes idmap config * : backend = tdb ------------------------------------------------------------------------ IPA ID Ranges: $ ipa idrange-find ---------------- 2 ranges matched ---------------- Range name: IPA.SJRB.AD_id_range First Posix ID of the range: 686600000 Number of IDs in the range: 200000 First RID of the corresponding RID range: 1000 First RID of the secondary RID range: 100000000 Range type: local domain range Range name: PRD.AD_id_range First Posix ID of the range: 342600000 Number of IDs in the range: 1200000 Domain SID of the trusted domain: S-1-5-21-81011375-1617023571-2284030129 Range type: Active Directory trust range with POSIX attributes ------------------------------------------------------------------------ IPA Server -> sssd.conf: [domain/ipa.sjrb.ad] cache_credentials = True krb5_store_password_if_offline = True ipa_domain = ipa.sjrb.ad id_provider = ipa auth_provider = ipa access_provider = ipa ipa_hostname = prdidml01.ipa.sjrb.ad chpass_provider = ipa ipa_server = prdidml01.ipa.sjrb.ad ipa_server_mode = True ldap_tls_cacert = /etc/ipa/ca.crt subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout, ldap_user_principal, ldap_use_tokengroups ignore_group_members = True ldap_purge_cache_timeout = 0 ldap_user_principal = nosuchattr ldap_use_tokengroups = False override_homedir = /home/%u entry_cache_timeout = 60 [sssd] services = pam, nss, ifp, sudo, ssh domains = ipa.sjrb.ad default_domain_suffix = sjrb.ad [nss] memcache_timeout = 30 homedir_substring = /home [ifp] allowed_uids = ipaapi, root ------------------------------------------------------------------------ IPA Client -> sssd.conf: [domain/ipa.sjrb.ad] cache_credentials = True krb5_store_password_if_offline = True krb5_auth_timeout = 30 krb5_validate = false ipa_domain = ipa.sjrb.ad id_provider = ipa auth_provider = ipa access_provider = ipa ldap_tls_cacert = /etc/ipa/ca.crt ipa_hostname = devntxappl60.sjrb.ad chpass_provider = ipa ipa_server = _srv_, prdidml01.ipa.sjrb.ad dns_discovery_domain = ipa.sjrb.ad ldap_sudo_smart_refresh_interval = 30 ldap_sudo_full_refresh_interval = 300 ipa_hbac_refresh = 5 entry_cache_timeout = 60 [sssd] services = nss, sudo, pam, ssh, pac domains = ipa.sjrb.ad default_domain_suffix = sjrb.ad [nss] homedir_substring = /home memcache_timeout = 10 [pam] pam_id_timeout = 10 ======================================================================== Version-Release number of selected component : IPA Servers: 389-ds-base-1.3.10.2-12.el7_9.x86_64 389-ds-base-libs-1.3.10.2-12.el7_9.x86_64 ipa-client-4.6.8-5.el7_9.7.x86_64 ipa-client-common-4.6.8-5.el7_9.7.noarch ipa-common-4.6.8-5.el7_9.7.noarch ipa-server-4.6.8-5.el7_9.7.x86_64 ipa-server-common-4.6.8-5.el7_9.7.noarch ipa-server-trust-ad-4.6.8-5.el7_9.7.x86_64 kernel-3.10.0-1160.25.1.el7.x86_64 kernel-3.10.0-1160.31.1.el7.x86_64 kernel-3.10.0-1160.36.2.el7.x86_64 kernel-devel-3.10.0-1160.25.1.el7.x86_64 kernel-devel-3.10.0-1160.31.1.el7.x86_64 kernel-devel-3.10.0-1160.36.2.el7.x86_64 kernel-headers-3.10.0-1160.36.2.el7.x86_64 kernel-tools-3.10.0-1160.36.2.el7.x86_64 kernel-tools-libs-3.10.0-1160.36.2.el7.x86_64 krb5-libs-1.15.1-50.el7.x86_64 krb5-pkinit-1.15.1-50.el7.x86_64 krb5-server-1.15.1-50.el7.x86_64 krb5-workstation-1.15.1-50.el7.x86_64 openldap-2.4.44-23.el7_9.x86_64 openldap-clients-2.4.44-23.el7_9.x86_64 samba-4.10.16-15.el7_9.x86_64 samba-client-4.10.16-15.el7_9.x86_64 samba-client-libs-4.10.16-15.el7_9.x86_64 samba-common-4.10.16-15.el7_9.noarch samba-common-libs-4.10.16-15.el7_9.x86_64 samba-common-tools-4.10.16-15.el7_9.x86_64 samba-libs-4.10.16-15.el7_9.x86_64 samba-python-4.10.16-15.el7_9.x86_64 samba-winbind-4.10.16-15.el7_9.x86_64 samba-winbind-modules-4.10.16-15.el7_9.x86_64 sssd-1.16.5-10.el7_9.8.x86_64 sssd-ad-1.16.5-10.el7_9.8.x86_64 sssd-client-1.16.5-10.el7_9.8.x86_64 sssd-common-1.16.5-10.el7_9.8.x86_64 sssd-common-pac-1.16.5-10.el7_9.8.x86_64 sssd-dbus-1.16.5-10.el7_9.8.x86_64 sssd-ipa-1.16.5-10.el7_9.8.x86_64 sssd-krb5-1.16.5-10.el7_9.8.x86_64 sssd-krb5-common-1.16.5-10.el7_9.8.x86_64 sssd-ldap-1.16.5-10.el7_9.8.x86_64 sssd-proxy-1.16.5-10.el7_9.8.x86_64 sssd-tools-1.16.5-10.el7_9.8.x86_64 IPA Client: adcli-0.8.2-9.el8.x86_64 ipa-client-4.9.2-3.module+el8.4.0+10413+a92f1bfa.x86_64 ipa-client-common-4.9.2-3.module+el8.4.0+10413+a92f1bfa.noarch ipa-common-4.9.2-3.module+el8.4.0+10413+a92f1bfa.noarch ipa-selinux-4.9.2-3.module+el8.4.0+10413+a92f1bfa.noarch kernel-4.18.0-305.3.1.el8_4.x86_64 kernel-4.18.0-305.10.2.el8_4.x86_64 kernel-core-4.18.0-305.3.1.el8_4.x86_64 kernel-core-4.18.0-305.10.2.el8_4.x86_64 kernel-modules-4.18.0-305.3.1.el8_4.x86_64 kernel-modules-4.18.0-305.10.2.el8_4.x86_64 kernel-tools-4.18.0-305.10.2.el8_4.x86_64 kernel-tools-libs-4.18.0-305.10.2.el8_4.x86_64 krb5-libs-1.18.2-8.el8.x86_64 krb5-workstation-1.18.2-8.el8.x86_64 openldap-2.4.46-17.el8_4.x86_64 realmd-0.16.3-22.el8.x86_64 samba-client-libs-4.13.3-3.el8.x86_64 samba-common-4.13.3-3.el8.noarch samba-common-libs-4.13.3-3.el8.x86_64 samba-common-tools-4.13.3-3.el8.x86_64 samba-libs-4.13.3-3.el8.x86_64 sssd-2.4.0-9.el8_4.1.x86_64 sssd-ad-2.4.0-9.el8_4.1.x86_64 sssd-client-2.4.0-9.el8_4.1.x86_64 sssd-common-2.4.0-9.el8_4.1.x86_64 sssd-common-pac-2.4.0-9.el8_4.1.x86_64 sssd-dbus-2.4.0-9.el8_4.1.x86_64 sssd-ipa-2.4.0-9.el8_4.1.x86_64 sssd-kcm-2.4.0-9.el8_4.1.x86_64 sssd-krb5-2.4.0-9.el8_4.1.x86_64 sssd-krb5-common-2.4.0-9.el8_4.1.x86_64 sssd-ldap-2.4.0-9.el8_4.1.x86_64 sssd-nfs-idmap-2.4.0-9.el8_4.1.x86_64 sssd-proxy-2.4.0-9.el8_4.1.x86_64 sssd-tools-2.4.0-9.el8_4.1.x86_64 ======================================================================== How reproducible: Issue is intermittent. Actual results: Intermittent login (in differents IPA clients, with differents users) Expected results: Users able to login Additional info: Case shared to IDM-tech [*] https://mailman-int.corp.redhat.com/archives/idm-tech/2021-September/msg00020.html