RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2006382 - IPA Intermittence fetching groups
Summary: IPA Intermittence fetching groups
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: sssd
Version: 7.9
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Sumit Bose
QA Contact: Jakub Vavra
URL:
Whiteboard: sync-to-jira
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-21 15:47 UTC by Chino Soliard
Modified: 2023-05-09 14:10 UTC (History)
19 users (show)

Fixed In Version: sssd-1.16.5-10.el7_9.12
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-02-22 17:03:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github SSSD sssd issues 5875 0 None closed CLDAP ping timeout is too long 2021-12-03 14:34:28 UTC
Red Hat Issue Tracker RHELPLAN-97768 0 None None None 2021-09-21 15:48:45 UTC
Red Hat Issue Tracker SSSD-4149 0 None Closed RFE RHEL-8+, postgresql-12+: Statistics needs same guarantees like plain database (currently reset zo zero when crash) 2022-06-27 06:14:33 UTC
Red Hat Product Errata RHBA-2022:0627 0 None None None 2022-02-22 17:04:08 UTC

Description Chino Soliard 2021-09-21 15:47:03 UTC
Description of problem:

IPA clients are having intermittent issues login in with AD user.
Environment is using HBAC and sometimes groups are not being fetched correctly
causing intermittent login issues.
Seems that Trust is failing.

Note: running 'ipa-adtrust-install' did not fix the issue.

Environment is:
- 4 IPA servers
- AD trust
- They are using POSIX attributes on AD groups

------------------------------------------------------------------------
On IPA Servers, we can see that:
- SSSD BE is going offline intermittently:
- Winbind is failing

    Sep 20 01:53:02 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab*****
    Sep 20 01:53:15 hostname smbd[PID]: [2021/09/20 01:53:15.070342,  0] ../../source3/auth/auth_winbind.c:143(check_winbind_security)
    Sep 20 01:53:15 hostname smbd[PID]:  check_winbind_security: winbindd not running - but required as DC with trusts: NT_STATUS_NO_LOGON_SERVERS
    Sep 20 01:53:42 hostname sssd[sssd]: Child [PID5] ('example.net':'%BE_example.net') was terminated by own WATCHDOG. Consult corresponding logs to figure out the reason.
    Sep 20 01:53:42 hostname sssd[be[example.net]]: Starting up
    Sep 20 01:53:42 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab*****
    Sep 20 01:53:42 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab*****
    Sep 20 01:53:42 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab*****
    Sep 20 01:53:42 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab*****
    Sep 20 01:53:45 hostname smbd[PID2]: [2021/09/20 01:53:45.115741,  0] ../../source3/auth/auth_winbind.c:143(check_winbind_security)
    Sep 20 01:53:45 hostname smbd[PID2]:  check_winbind_security: winbindd not running - but required as DC with trusts: NT_STATUS_NO_LOGON_SERVERS
    Sep 20 01:54:05 hostname smbd[PID3]: [2021/09/20 01:54:05.125284,  0] ../../source3/auth/auth_winbind.c:143(check_winbind_security)
    Sep 20 01:54:05 hostname smbd[PID3]:  check_winbind_security: winbindd not running - but required as DC with trusts: NT_STATUS_NO_LOGON_SERVERS
    Sep 20 01:54:25 hostname smbd[PID4]: [2021/09/20 01:54:25.162790,  0] ../../source3/auth/auth_winbind.c:143(check_winbind_security)
    Sep 20 01:54:25 hostname smbd[PID4]:  check_winbind_security: winbindd not running - but required as DC with trusts: NT_STATUS_NO_LOGON_SERVERS
    Sep 20 01:55:33 hostname sssd[sssd]: Child [PID6] ('example.net':'%BE_example.net') was terminated by own WATCHDOG. Consult corresponding logs to figure out the reason.
    Sep 20 01:55:33 hostname sssd[be[example.net]]: Starting up
    Sep 20 01:55:33 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab*****
    Sep 20 01:55:33 hostname sssd: Keytab successfully retrieved and stored in: /var/lib/sss/keytabs/PRD.AD.keytab*****

Note: In this particular moment, both services failed together, but they commonly fail lonely.

------------------------------------------------------------------------
Here's the failure on the IPA Client to look up the AD User:

    (2021-08-26  8:16:34): [be[example.net]] [ipa_s2n_get_acct_info_send] (0x0400): Sending request_type: [REQ_FULL_WITH_MEMBERS] for trust user [ADuser] to IPA server
    (2021-08-26  8:16:34): [be[example.net]] [ipa_s2n_exop_send] (0x0400): Executing extended operation
    (2021-08-26  8:16:34): [be[example.net]] [ipa_s2n_exop_send] (0x2000): ldap_extended_operation sent, msgid = 109
    (2021-08-26  8:16:34): [be[example.net]] [sdap_op_add] (0x2000): New operation 109 timeout 6
    (2021-08-26  8:16:34): [be[example.net]] [sdap_process_result] (0x2000): Trace: sh[0x5561682f4850], connected[1], ops[0x5561682dd2a0], ldap[0x55616827c120]
    (2021-08-26  8:16:34): [be[example.net]] [sdap_process_result] (0x2000): Trace: end of ldap_result list
    (2021-08-26  8:16:36): [be[example.net]] [sdap_process_result] (0x2000): Trace: sh[0x5561682f4850], connected[1], ops[0x5561682dd2a0], ldap[0x55616827c120]
    (2021-08-26  8:16:36): [be[example.net]] [sdap_process_message] (0x4000): Message type: [LDAP_RES_EXTENDED]
    (2021-08-26  8:16:36): [be[example.net]] [ipa_s2n_exop_done] (0x0040): ldap_extended_operation result: Time limit exceeded(3), (null).
    (2021-08-26  8:16:36): [be[example.net]] [ipa_s2n_exop_done] (0x0040): ldap_extended_operation failed, server logs might contain more details.
    (2021-08-26  8:16:36): [be[example.net]] [sdap_op_destructor] (0x2000): Operation 109 finished
    (2021-08-26  8:16:36): [be[example.net]] [ipa_s2n_get_user_done] (0x0040): s2n exop request failed.
    (2021-08-26  8:16:36): [be[example.net]] [sdap_id_op_done] (0x4000): releasing operation connection
    (2021-08-26  8:16:36): [be[example.net]] [ipa_subdomain_account_done] (0x0040): ipa_get_*_acct request failed: [1432158230]: Network I/O Error.

Seems like a trust failure (s2n exop request failed)

------------------------------------------------------------------------
IPA Servers -> Samba debug logs -> log.winbind-idmap:
    [2021/09/08 08:54:07.763659,  1] ../../source3/winbindd/idmap_tdb_common.c:66(idmap_tdb_common_allocate_id_action)
      Fatal Error: GID range full!! (max: 0)
    [2021/09/08 08:54:07.763682,  1] ../../source3/winbindd/idmap_tdb_common.c:138(idmap_tdb_common_allocate_id)
      Error allocating a new GID

------------------------------------------------------------------------
SAMBA config:

    [global]
        create krb5 conf = No
        dedicated keytab file = /etc/samba/samba.keytab
        disable spoolss = Yes
        domain logons = Yes
        domain master = Yes
        kerberos method = dedicated keytab
        ldap group suffix = cn=groups,cn=accounts
        ldap machine suffix = cn=computers,cn=accounts
        ldap ssl = no
        ldap suffix = dc=ipa,dc=sjrb,dc=ad
        ldap user suffix = cn=users,cn=accounts
        log file = /var/log/samba/log.%m
        max log size = 100000
        max smbd processes = 1000
        passdb backend = ipasam:ldapi://%2fvar%2frun%2fslapd-IPA-SJRB-AD.socket
        realm = IPA.SJRB.AD
        registry shares = Yes
        security = USER
        workgroup = IPA
        idmap config ipa : range = 686600000 - 686800000
        idmap config ipa : backend = sss
        idmap config * : range = 0 - 0
        rpc_daemon:lsasd = fork
        rpc_daemon:epmd = fork
        rpc_server:tcpip = yes
        rpc_server:netlogon = external
        rpc_server:samr = external
        rpc_server:lsasd = external
        rpc_server:lsass = external
        rpc_server:lsarpc = external
        rpc_server:epmapper = external
        ldapsam:trusted = yes
        idmap config * : backend = tdb

------------------------------------------------------------------------
IPA ID Ranges:

    $ ipa idrange-find
    ----------------
    2 ranges matched
    ----------------
      Range name: IPA.SJRB.AD_id_range
      First Posix ID of the range: 686600000
      Number of IDs in the range: 200000
      First RID of the corresponding RID range: 1000
      First RID of the secondary RID range: 100000000
      Range type: local domain range
    
      Range name: PRD.AD_id_range
      First Posix ID of the range: 342600000
      Number of IDs in the range: 1200000
      Domain SID of the trusted domain: S-1-5-21-81011375-1617023571-2284030129
      Range type: Active Directory trust range with POSIX attributes
    
------------------------------------------------------------------------

IPA Server -> sssd.conf:

    [domain/ipa.sjrb.ad]
    cache_credentials = True
    krb5_store_password_if_offline = True
    ipa_domain = ipa.sjrb.ad
    id_provider = ipa
    auth_provider = ipa
    access_provider = ipa
    ipa_hostname = prdidml01.ipa.sjrb.ad
    chpass_provider = ipa
    ipa_server = prdidml01.ipa.sjrb.ad
    ipa_server_mode = True
    ldap_tls_cacert = /etc/ipa/ca.crt
    subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout, ldap_user_principal, ldap_use_tokengroups
    ignore_group_members = True
    ldap_purge_cache_timeout = 0
    ldap_user_principal = nosuchattr
    ldap_use_tokengroups = False
    override_homedir = /home/%u
    entry_cache_timeout = 60

    [sssd]
    services = pam, nss, ifp, sudo, ssh
    domains = ipa.sjrb.ad
    default_domain_suffix = sjrb.ad

    [nss]
    memcache_timeout = 30
    homedir_substring = /home

    [ifp]
    allowed_uids = ipaapi, root

------------------------------------------------------------------------

IPA Client -> sssd.conf:


    [domain/ipa.sjrb.ad]
    cache_credentials = True
    krb5_store_password_if_offline = True
    krb5_auth_timeout = 30
    krb5_validate = false
    ipa_domain = ipa.sjrb.ad
    id_provider = ipa
    auth_provider = ipa
    access_provider = ipa
    ldap_tls_cacert = /etc/ipa/ca.crt
    ipa_hostname = devntxappl60.sjrb.ad
    chpass_provider = ipa
    ipa_server = _srv_, prdidml01.ipa.sjrb.ad
    dns_discovery_domain = ipa.sjrb.ad
    ldap_sudo_smart_refresh_interval = 30
    ldap_sudo_full_refresh_interval = 300
    ipa_hbac_refresh = 5
    entry_cache_timeout = 60

    [sssd]
    services = nss, sudo, pam, ssh, pac
    domains = ipa.sjrb.ad
    default_domain_suffix = sjrb.ad

    [nss]
    homedir_substring = /home
    memcache_timeout = 10

    [pam]
    pam_id_timeout = 10

========================================================================

Version-Release number of selected component :
IPA Servers:
    389-ds-base-1.3.10.2-12.el7_9.x86_64
    389-ds-base-libs-1.3.10.2-12.el7_9.x86_64
    ipa-client-4.6.8-5.el7_9.7.x86_64
    ipa-client-common-4.6.8-5.el7_9.7.noarch
    ipa-common-4.6.8-5.el7_9.7.noarch
    ipa-server-4.6.8-5.el7_9.7.x86_64
    ipa-server-common-4.6.8-5.el7_9.7.noarch
    ipa-server-trust-ad-4.6.8-5.el7_9.7.x86_64
    kernel-3.10.0-1160.25.1.el7.x86_64
    kernel-3.10.0-1160.31.1.el7.x86_64
    kernel-3.10.0-1160.36.2.el7.x86_64
    kernel-devel-3.10.0-1160.25.1.el7.x86_64
    kernel-devel-3.10.0-1160.31.1.el7.x86_64
    kernel-devel-3.10.0-1160.36.2.el7.x86_64
    kernel-headers-3.10.0-1160.36.2.el7.x86_64
    kernel-tools-3.10.0-1160.36.2.el7.x86_64
    kernel-tools-libs-3.10.0-1160.36.2.el7.x86_64
    krb5-libs-1.15.1-50.el7.x86_64
    krb5-pkinit-1.15.1-50.el7.x86_64
    krb5-server-1.15.1-50.el7.x86_64
    krb5-workstation-1.15.1-50.el7.x86_64
    openldap-2.4.44-23.el7_9.x86_64
    openldap-clients-2.4.44-23.el7_9.x86_64
    samba-4.10.16-15.el7_9.x86_64
    samba-client-4.10.16-15.el7_9.x86_64
    samba-client-libs-4.10.16-15.el7_9.x86_64
    samba-common-4.10.16-15.el7_9.noarch
    samba-common-libs-4.10.16-15.el7_9.x86_64
    samba-common-tools-4.10.16-15.el7_9.x86_64
    samba-libs-4.10.16-15.el7_9.x86_64
    samba-python-4.10.16-15.el7_9.x86_64
    samba-winbind-4.10.16-15.el7_9.x86_64
    samba-winbind-modules-4.10.16-15.el7_9.x86_64
    sssd-1.16.5-10.el7_9.8.x86_64
    sssd-ad-1.16.5-10.el7_9.8.x86_64
    sssd-client-1.16.5-10.el7_9.8.x86_64
    sssd-common-1.16.5-10.el7_9.8.x86_64
    sssd-common-pac-1.16.5-10.el7_9.8.x86_64
    sssd-dbus-1.16.5-10.el7_9.8.x86_64
    sssd-ipa-1.16.5-10.el7_9.8.x86_64
    sssd-krb5-1.16.5-10.el7_9.8.x86_64
    sssd-krb5-common-1.16.5-10.el7_9.8.x86_64
    sssd-ldap-1.16.5-10.el7_9.8.x86_64
    sssd-proxy-1.16.5-10.el7_9.8.x86_64
    sssd-tools-1.16.5-10.el7_9.8.x86_64

IPA Client: 

    adcli-0.8.2-9.el8.x86_64
    ipa-client-4.9.2-3.module+el8.4.0+10413+a92f1bfa.x86_64
    ipa-client-common-4.9.2-3.module+el8.4.0+10413+a92f1bfa.noarch
    ipa-common-4.9.2-3.module+el8.4.0+10413+a92f1bfa.noarch
    ipa-selinux-4.9.2-3.module+el8.4.0+10413+a92f1bfa.noarch
    kernel-4.18.0-305.3.1.el8_4.x86_64
    kernel-4.18.0-305.10.2.el8_4.x86_64
    kernel-core-4.18.0-305.3.1.el8_4.x86_64
    kernel-core-4.18.0-305.10.2.el8_4.x86_64
    kernel-modules-4.18.0-305.3.1.el8_4.x86_64
    kernel-modules-4.18.0-305.10.2.el8_4.x86_64
    kernel-tools-4.18.0-305.10.2.el8_4.x86_64
    kernel-tools-libs-4.18.0-305.10.2.el8_4.x86_64
    krb5-libs-1.18.2-8.el8.x86_64
    krb5-workstation-1.18.2-8.el8.x86_64
    openldap-2.4.46-17.el8_4.x86_64
    realmd-0.16.3-22.el8.x86_64
    samba-client-libs-4.13.3-3.el8.x86_64
    samba-common-4.13.3-3.el8.noarch
    samba-common-libs-4.13.3-3.el8.x86_64
    samba-common-tools-4.13.3-3.el8.x86_64
    samba-libs-4.13.3-3.el8.x86_64
    sssd-2.4.0-9.el8_4.1.x86_64
    sssd-ad-2.4.0-9.el8_4.1.x86_64
    sssd-client-2.4.0-9.el8_4.1.x86_64
    sssd-common-2.4.0-9.el8_4.1.x86_64
    sssd-common-pac-2.4.0-9.el8_4.1.x86_64
    sssd-dbus-2.4.0-9.el8_4.1.x86_64
    sssd-ipa-2.4.0-9.el8_4.1.x86_64
    sssd-kcm-2.4.0-9.el8_4.1.x86_64
    sssd-krb5-2.4.0-9.el8_4.1.x86_64
    sssd-krb5-common-2.4.0-9.el8_4.1.x86_64
    sssd-ldap-2.4.0-9.el8_4.1.x86_64
    sssd-nfs-idmap-2.4.0-9.el8_4.1.x86_64
    sssd-proxy-2.4.0-9.el8_4.1.x86_64
    sssd-tools-2.4.0-9.el8_4.1.x86_64

========================================================================

How reproducible:

   Issue is intermittent.

Actual results:

   Intermittent login (in differents IPA clients, with differents users)

Expected results:

   Users able to login

Additional info:
  Case shared to IDM-tech
  [*] https://mailman-int.corp.redhat.com/archives/idm-tech/2021-September/msg00020.html

Comment 25 Theodoros Apazoglou 2021-11-03 14:44:20 UTC
Moving this BZ from IPA to SSSD as proposed by Sumit on https://bugzilla.redhat.com/show_bug.cgi?id=2006382#c18

Comment 32 Alexey Tikhonov 2021-12-03 14:36:46 UTC
https://github.com/SSSD/sssd/pull/5883 was merged upstream, but not yet to 1-16 branch.

Comment 36 Alexey Tikhonov 2021-12-10 17:29:05 UTC
1-16 backport: https://github.com/SSSD/sssd/pull/5909

Pushed PR: https://github.com/SSSD/sssd/pull/5909

* `sssd-1-16`
    * be3ee30c68dd9d2e5184da226dfbe66f516a4b92 - cldap: use dns_resolver_server_timeout timeout for cldap ping
    * ed243335d3e74ab2cde49eacc9a85ca5408a8dec - ad: only send cldap-ping to our local domain
    * 664ae9d2247b5139d2286975228baa0cea39a8e4 - ad: make ad_srv_plugin_ctx_switch_site() public
    * f60a6fc682646a8c16fa8875456300c61cf3e979 - ad: use already discovered forest name
    * ecfb7df52fd3b0edf8549d42cfa6b378407fb982 - ad: move current site and forest name to a more global context
    * 4b59b0af3d97b2a6b0acc08fa80377a5f59e5bfe - ad: require name when looking up root domain
    * 4b02c9632c44265124e4fa130170bc86de369b8c - SSSD man: man_dns_resolver_parameter_modification

Comment 56 errata-xmlrpc 2022-02-22 17:03:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sssd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0627


Note You need to log in before you can comment on or make changes to this bug.