RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1945644 - systemd provider in /etc/nsswitch.conf introduces 25s lockups for NFS share mount
Summary: systemd provider in /etc/nsswitch.conf introduces 25s lockups for NFS share m...
Keywords:
Status: CLOSED DUPLICATE of bug 1694681
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: selinux-policy
Version: 8.2
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: beta
: ---
Assignee: Zdenek Pytela
QA Contact: BaseOS QE Security Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-01 14:01 UTC by Leonid Titov
Modified: 2022-01-05 13:43 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-04-08 07:08:05 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1694681 1 medium CLOSED Slow listing of files owned by 'nobody' on nfs share with SELinux and 'filter_users' 2024-10-01 16:16:42 UTC

Description Leonid Titov 2021-04-01 14:01:50 UTC
Description of problem:
NFS mount can be locked up to 25 seconds while looking up for non-existing user:

Mar 29 03:40:38 <hostname> nfsidmap[3348432]: libnfsidmap: res_querydomain() failed for _nfsv4idmapdomain.<domain>: Unknown host
Mar 29 03:40:38 <hostname> nfsidmap[3348432]: libnfsidmap: using (default) domain: <domain>
Mar 29 03:40:38 <hostname> nfsidmap[3348432]: libnfsidmap: Realms list: '<domain>'
Mar 29 03:40:38 <hostname> nfsidmap[3348432]: libnfsidmap: loaded plugin /usr/lib64/libnfsidmap/nsswitch.so for method nsswitch
Mar 29 03:41:03 <hostname> nfsidmap[3348432]: nss_getpwnam: name '<user>' not found in domain '<domain>'

- you see 25s gap between latest two messages

if /etc/nsswitch.conf contains 'systemd' passwd provider:

passwd: sss files altfiles systemd

This delay disappears if we remove systemd out of nsswitch.conf:

Mar 30 15:00:42 <hostname> nfsidmap[3569031]: libnfsidmap: loaded plugin /usr/lib64/libnfsidmap/nsswitch.so for method nsswitch
Mar 30 15:00:42 <hostname> nfsidmap[3569031]: nss_getpwnam: name '<user>' not found in domain '<domain>'

- 'user not found' message appears without delay

Version-Release number of selected component (if applicable):
OpenShift Container Platform 4.6, 4.7

How reproducible:

Steps to Reproduce:
1. Try to mount NFS V4 share with non-existing user

2. Observe long delay

3. Remove systemd provider from nsswitch.conf

4. Observe fast mount


Additional info:
Looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1694681
but keeping in mind that we don't use local users in CoreOS, maybe it will be better to just remove systemd provider from nsswitch.conf

Comment 1 Micah Abbott 2021-04-01 15:03:06 UTC
Looking at BZ#1694681, the symptoms look very similar.

For confirmation, is are there any audit messages in /var/log/audit.log on the node?  Or reported via `ausearch -m avc`?

If there is a denial similar to what is reported in https://bugzilla.redhat.com/show_bug.cgi?id=1694681#c17, it seems like this would be the same problem.

```
type=USER_AVC msg=audit(1589400931.223:5344): pid=993 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=error error_name=org.freedesktop.systemd1.NoSuchDynamicUser dest=:1.4536 spid=1 tpid=2770 scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:kernel_t:s0 tclass=dbus permissive=0  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'UID="dbus" AUID="unset" SAUID="dbus"
```

That BZ appears to be fixed in RHEL 8.4; if we want it fixed for OCP 4.6, we would need to request a backport all the way to RHEL 8.2 EUS

Comment 2 Leonid Titov 2021-04-02 09:11:01 UTC
Hello,

Unfortunately I don't see logged failure in audit logs:

$grep dbusd_t  var/log/audit/audit.log | wc -l
0

Comment 3 Micah Abbott 2021-04-06 19:53:24 UTC
You can try to workaround the problem by dropping a custom `/etc/nsswitch.conf` via Machine config like so:

```
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-nsswitch-remove-systemd
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
          - path: /etc/nsswitch.conf
            contents:
              source: data:text/plain;charset=utf-8;base64,IyBXcml0dGVuIHZpYSBNYWNoaW5lQ29uZmlnIDk5LW5zc3N3aXRjaC1yZW1vdmUtc3lzdGVtZApwYXNzd2Q6IHNzcyBmaWxlcyBhbHRmaWxlcwpncm91cDogc3NzIGZpbGVzIGFsdGZpbGVzIHN5c3RlbWQKbmV0Z3JvdXA6ICAgc3NzIGZpbGVzCmF1dG9tb3VudDogIHNzcyBmaWxlcwpzZXJ2aWNlczogICBzc3MgZmlsZXMKc2hhZG93OiAgICAgZmlsZXMgc3NzCmhvc3RzOiAgICAgIGZpbGVzIGRucyBteWhvc3RuYW1lCmFsaWFzZXM6ICAgIGZpbGVzCmV0aGVyczogICAgIGZpbGVzCmdzaGFkb3c6ICAgIGZpbGVzCm5ldHdvcmtzOiAgIGZpbGVzIGRucwpwcm90b2NvbHM6ICBmaWxlcwpwdWJsaWNrZXk6ICBmaWxlcwpycGM6ICAgICAgICBmaWxlcwoK
            user:
              name: root
            mode: 0644
```

I took the default `/etc/nsswitch.conf` from a node on OCP 4.7, stripped out all the comments, and removed `systemd` from the `passwd:` line.

```
$ echo "IyBXcml0dGVuIHZpYSBNYWNoaW5lQ29uZmlnIDk5LW5zc3N3aXRjaC1yZW1vdmUtc3lzdGVtZApwYXNzd2Q6IHNzcyBmaWxlcyBhbHRmaWxlcwpncm91cDogc3NzIGZpbGVzIGFsdGZpbGVzIHN5c3RlbWQKbmV0Z3JvdXA6ICAgc3NzIGZpbGVzCmF1dG9tb3VudDogIHNzcyBmaWxlcwpzZXJ2aWNlczogICBzc3MgZmlsZXMKc2hhZG93OiAgICAgZmlsZXMgc3NzCmhvc3RzOiAgICAgIGZpbGVzIGRucyBteWhvc3RuYW1lCmFsaWFzZXM6ICAgIGZpbGVzCmV0aGVyczogICAgIGZpbGVzCmdzaGFkb3c6ICAgIGZpbGVzCm5ldHdvcmtzOiAgIGZpbGVzIGRucwpwcm90b2NvbHM6ICBmaWxlcwpwdWJsaWNrZXk6ICBmaWxlcwpycGM6ICAgICAgICBmaWxlcwoK" | base64 -d
# Written via MachineConfig 99-nssswitch-remove-systemd
passwd: sss files altfiles
group: sss files altfiles systemd
netgroup:   sss files
automount:  sss files
services:   sss files
shadow:     files sss
hosts:      files dns myhostname
aliases:    files
ethers:     files
gshadow:    files
networks:   files dns
protocols:  files
publickey:  files
rpc:        files
```

And I'm going to send this over to the RHEL team for further investigation.  Please be prepared to provide additional information from affected nodes and to reproduce the problem.

Comment 7 Sumit Bose 2021-04-07 06:29:59 UTC
Hi,

can you check the output of the sesearch commands from https://bugzilla.redhat.com/show_bug.cgi?id=1694681#c24 and https://bugzilla.redhat.com/show_bug.cgi?id=1694681#c33 and change to component of this ticket to selinux-policy is the sesearch output indicates that it is the same issue?

bye,
Sumit

Comment 8 Leonid Titov 2021-04-07 10:57:28 UTC
@sbose here they are:

sh-5.0# sesearch --dontaudit -s init_t -t kernel_t -c dbus -p send_msg
dontaudit init_t kernel_t:dbus send_msg;

sh-5.0# sesearch -A -s init_t -t kernel_t -c dbus -p send_msg
sh-5.0# sesearch -A -t init_t -s kernel_t -c dbus -p send_msg
allow dbusd_unconfined init_t:dbus send_msg;
allow dbusd_unconfined nsswitch_domain:dbus send_msg;

Comment 9 Sumit Bose 2021-04-07 12:33:39 UTC
(In reply to Leonid Titov from comment #8)
> @sbose here they are:
> 
> sh-5.0# sesearch --dontaudit -s init_t -t kernel_t -c dbus -p send_msg
> dontaudit init_t kernel_t:dbus send_msg;
> 
> sh-5.0# sesearch -A -s init_t -t kernel_t -c dbus -p send_msg
> sh-5.0# sesearch -A -t init_t -s kernel_t -c dbus -p send_msg
> allow dbusd_unconfined init_t:dbus send_msg;
> allow dbusd_unconfined nsswitch_domain:dbus send_msg;

Hi,

thanks, this really looks like https://bugzilla.redhat.com/show_bug.cgi?id=1694681, I'll change the component to selinux-policy. Maybe the selinux maintainers even know a workaround to modify the running policy to allow those operations.

bye,
Sumit

Comment 10 Zdenek Pytela 2021-04-07 12:47:04 UTC
I can confirm the symptoms look the same, hence I am about to close this bz a duplicate. If a fix for EUS is required, please follow the acceleration fixes workflow.

To work around the problem, a local selinux policy module can be installed:

  # cat local_init_kernel_dbus.cil
(allow init_t kernel_t (dbus (send_msg)))
  # semodule -i local_init_kernel_dbus.cil

Comment 11 Zdenek Pytela 2021-04-08 07:08:05 UTC

*** This bug has been marked as a duplicate of bug 1694681 ***


Note You need to log in before you can comment on or make changes to this bug.