Bug 1827432

Summary: When the passwd or group files are replaced, sssd stops monitoring the file for inotify events, and no updates are triggered
Product: Red Hat Enterprise Linux 8 Reporter: Frank Sorenson <fsorenso>
Component: sssdAssignee: Alexey Tikhonov <atikhono>
Status: CLOSED ERRATA QA Contact: sssd-qe <sssd-qe>
Severity: low Docs Contact:
Priority: low    
Version: 8.1CC: aborah, abroy, atikhono, dlavu, grajaiya, jhrozek, jorge.fabregas, lslebodn, mzidek, pbrezina, sgoveas, tscherf
Target Milestone: rcKeywords: Reproducer, Triaged
Target Release: 8.0Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: sync-to-jira review
Fixed In Version: sssd-2.3.0-3.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-04 02:05:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Frank Sorenson 2020-04-23 22:10:56 UTC 
Description of problem:

When the /etc/passwd or /etc/group file is replaced by another file, sssd no longer monitors that file for inotify events, and reloads no longer occur when the files are modified.


When the file is modified in-place, the events are still received, and reloads still occur.


Version-Release number of selected component (if applicable):

	sssd-2.0.0-43.el8.x86_64


How reproducible:

	easy, see below


Steps to Reproduce:


	(the customer's use case included modifying with 'vi', however the following is a scriptable reproducer of the same)


run sssd with debugging to watch for when inotify events occur (debug-level 0x3400 appears to log all of the needed debug messages related to the inotify events):

	# /usr/sbin/sssd -i --logger=stderr --debug-level=0x3400 2>&1 | tee /var/tmp/out
	...


	# getent group g10000
	# echo "g10000:x:10000: >> /etc/group
	# getent group g10000
	g10000:x:10000:

(sssd reloads /etc/group)

	# cp /etc/group /etc/group.new
	# echo g10001:x:10001: >> /etc/group.tmp
	# mv -f /etc/group.tmp /etc/group

(sssd reloads /etc/group)

	# echo g10002:x:10002: >> /etc/group

(sssd does not reload /etc/group)


can also test with:
	# touch /etc/group
	(because no inotify watch exists for this file, no reload is triggered)


Actual results:

	/etc/group is not immediately reloaded


Expected results:

	/etc/group is reloaded when modified


Additional info:


To correct the bug, when the inotify watch on /etc detects that /etc/passwd or /etc/group has been replaced, the watch for the old inode needs to be removed, and a new watch should be added for the new file.


when the watched file is replaced, the following inotify events may be generated *for the watch on the replaced file*

	<Event dir=False mask=0x4 maskname=IN_ATTRIB name='' path=/etc/group pathname=/etc/group wd=1 >
	<Event dir=False mask=0x400 maskname=IN_DELETE_SELF name='' path=/etc/group pathname=/etc/group wd=1 >
	<Event dir=False mask=0x8000 maskname=IN_IGNORED name='' path=/etc/group pathname=/etc/group wd=1 >


and the following inotify events may be generated *for the watch on the /etc directory*

	<Event cookie=32102 dir=False mask=0x40 maskname=IN_MOVED_FROM name=group.new path=/etc pathname=/etc/group.new wd=1 >
	<Event cookie=32102 dir=False mask=0x80 maskname=IN_MOVED_TO name=group path=/etc pathname=/etc/group src_pathname=/etc/group.new wd=1 >
	<Event dir=False mask=0x20 maskname=IN_OPEN name=group path=/etc pathname=/etc/group wd=1 >
	<Event dir=False mask=0x1 maskname=IN_ACCESS name=group path=/etc pathname=/etc/group wd=1 >
	<Event dir=False mask=0x10 maskname=IN_CLOSE_NOWRITE name=group path=/etc pathname=/etc/group wd=1 >


currently only these events are enabled and received for the file:
	<Event dir=False mask=0x400 maskname=IN_DELETE_SELF name='' path=/etc/group pathname=/etc/group wd=1 >
	<Event dir=False mask=0x8000 maskname=IN_IGNORED name='' path=/etc/group pathname=/etc/group wd=1 >

and this event for the directory:
	<Event cookie=32102 dir=False mask=0x80 maskname=IN_MOVED_TO name=group path=/etc pathname=/etc/group wd=1 >
Comment 1 Frank Sorenson 2020-04-23 23:00:41 UTC 
Customer's specific issue:

add a user to a group by replacing the /etc/group file:
	# u=foo ; sed -e "s@^\(wheel:.\+[^:]\)\$@\1,$u@g" -e "s/^\(wheel:.\+:\)\$/\1$u/g" /etc/group >/var/tmp/group.new
	# mv -f /var/tmp/group.new /etc/group

	# getent group wheel
	wheel:x:10:foo

	# grep wheel /etc/group
	wheel:x:10:foo


now add a user by replacing just the file contents:
	# u=bar ; sed -e "s@^\(wheel:.\+[^:]\)\$@\1,$u@g" -e "s/^\(wheel:.\+:\)\$/\1$u/g" /etc/group >/var/tmp/group.new
	# cat /var/tmp/group.new >/etc/group
	# rm -f /var/tmp/group.new

	# getent group wheel
	wheel:x:10:foo

	# grep wheel /etc/group
	wheel:x:10:foo,bar

because there was no reload, the group does not include the new user.


after restarting sssd:
	# systemctl restart sssd.service
	# getent group wheel
	wheel:x:10:foo,bar




to see the currently-watched events generated on /etc/group:
	# pyinotify -e IN_CLOSE_WRITE,IN_MOVED_TO,IN_CREATE,IN_DELETE_SELF,IN_MOVE_SELF /etc/group

to see the currently-watched events generated on /etc
	# pyinotify -e IN_MOVED_TO,IN_CREATE /etc


after the /etc/passwd or /etc/group file is replaced, no further notifications are received for the file


to see all events:
	# pyinotify /etc
Comment 4 Frank Sorenson 2020-04-24 15:34:25 UTC 
just a quick note that 'pyinotify' is the RHEL 8 replacement for inotifywatch/inotifywait, and is delivered in the python3-inotify rpm:

	python3-inotify-0.9.6-13.el8.noarch

# pyinotify -h
Usage: pyinotify [options] [path1] [path2] [pathn]

Options:
  -h, --help            show this help message and exit
  -v, --verbose         Verbose mode
  -r, --recursive       Add watches recursively on paths
  -a, --auto_add        Automatically add watches on new directories
  -g, --glob            Treat paths as globs
  -e EVENT[,...], --events-list=EVENT[,...]
                        A comma-separated list of events to watch for - see
                        the documentation for valid options (defaults to
                        everything)
  -s, --stats           Display dummy statistics
  -V, --version         Pyinotify version
  -f, --raw-format      Disable enhanced output format.
  -c COMMAND, --command=COMMAND
                        Shell command to run upon event
Comment 12 Jorge Fábregas 2020-05-01 20:25:56 UTC 
I'm the "customer" who reported this.  In a nutshell, this is the way to easily recreate the issue on a fresh RHEL8 installation:

1) umask 027
2) create new user
3) modify /etc/groiup via "vi" to include that new user to a whatever group
4) check with "getent group whateverGroup"

You'll see it won't show up.  I specified a mask of 027 because that's what our system had (a deviation from the standard 002).   It turns out that "vim" behaves differently based on the current mask.  If you have a mask of 027 you'll experience what's described by Frank.

We know the best practice for changing group ownerships is via "usermod" or "vigr" but, this a Linux system and a plain text file (/etc/group) so, in that spirit, many admins do modify the file by hand for simple edits and so.  Hope this was helpful.

-- 
Jorge
Comment 13 Alexey Tikhonov 2020-06-12 20:21:58 UTC 
Part of upstream PR https://github.com/SSSD/sssd/pull/5204
Comment 14 Pavel Březina 2020-06-18 10:19:42 UTC 
Pushed PR: https://github.com/SSSD/sssd/pull/5204

* `master`
    * 5611d242f7d958ec06e07f601eb9f2e9a1a6b696 - PROVIDERS/IPA: fixed few CLANG_WARNINGs
    * ce06995432158ded2f95017c6b512c5a4a85f4a9 - PROVIDERS/LDAP: fixed CLANG_WARNING
    * 64adcd410b889f5cbdb39e6e18a9a24f3a61f9ee - PROVIDERS/LDAP: fixed CLANG_WARNING
    * 316c850ec3daf914b6c76f1e0f4a2f0dcae6c4c9 - CACHE_REQ: fixed CLANG_WARNING
    * 83389697f62e6cc2ce361999c194c555dc1464ab - RESPONDER/NSS: fixed few CLANG_WARNINGs
    * 464f809e00d663cda980a2329c93286c1f599090 - RESPONDER/SUDO: fixed CLANG_WARNING
    * 14e5c31e582943330289c6dac3753033dc41d4c7 - CLIENT: fixed "Dereference of null pointer" warning
    * e525ed6a6acd2bcb6bae6967e6e6aaaa1c8b19f9 - TOOLS: fixed a couple of CLANG_WARNINGs
    * 9c4d662de64d483bab3e2cdca5cab8e9b3a88457 - TOOLS: fixed CLANG_WARNING
    * 0c5711f9bae1cb46d4cd3fbe5d86d8688087be13 - util/inotify: fixed bug in inotify event processing
    * 144e78dfebc0fd01feb6c11a37f81d01146cf33a - util/inotify: fixed CLANG_WARNING
    * 6701ad96ac1ddd49998fe14e6b3c6c26a1e88944 - CLIENT: fixed CPPCHECK_WARNING (CWE-476)
    * b132fab8c0fb03700a52d7e1b6577ea1cc043179 - MC: fixed CPPCHECK_WARNING
    * 8088b3e3adefc0dc96af501bae5bba6e2e1f8a91 - PROXY: suppress CPPCHECK_WARNING (CWE-456)
    * ee16f3928d948e9a9a619d62136a25c5b73cd009 - KCM: fixed NO_EFFECT (CWE-398)
    * 014cbde8fce1571dc4bd161b8d50eaaad5b865b6 - NSS: fixed FORWARD_NULL (CWE-476)
    * 39480618ae89138b1dc078229a5f1e58bc88c0eb - CLIENT: fixed few CHECKED_RETURN (CWE-252) warnings

This bug is probably fixed only by "0c5711f9bae1cb46d4cd3fbe5d86d8688087be13 - util/inotify: fixed bug in inotify event processing" but I'm not sure if it depends on the other patches or not.
Comment 16 Alexey Tikhonov 2020-06-26 17:14:27 UTC 
Only relevant patches are:
 * 0c5711f9bae1cb46d4cd3fbe5d86d8688087be13 - util/inotify: fixed bug in inotify event processing
 * 144e78dfebc0fd01feb6c11a37f81d01146cf33a - util/inotify: fixed CLANG_WARNING
Comment 19 Anuj Borah 2020-08-11 04:55:10 UTC 
First comment: This bug only occures when umask is set to 027
===========================================
[root@ci-vm-10-0-139-169 etc]# rpm -qa | grep sssd
sssd-nfs-idmap-2.0.0-43.el8.x86_64
sssd-client-2.0.0-43.el8.x86_64
sssd-common-2.0.0-43.el8.x86_64
sssd-kcm-2.0.0-43.el8.x86_64
[root@ci-vm-10-0-139-169 etc]# umask 027
[root@ci-vm-10-0-139-169 etc]# useradd anuj3
[root@ci-vm-10-0-139-169 etc]# vi group(manually add user anuj3 to group wheel)
[root@ci-vm-10-0-139-169 etc]# getent group wheel
wheel:x:10:anuj,anuj1
[root@ci-vm-10-0-139-169 etc]# grep wheel group
wheel:x:10:anuj,anuj1,anuj3
[root@ci-vm-10-0-139-169 etc]# pwd
/etc
==============================================
[root@ci-vm-10-0-138-127 etc]# rpm -qa | grep sssd
sssd-client-2.3.0-6.el8.x86_64
sssd-nfs-idmap-2.3.0-6.el8.x86_64
python3-sssdconfig-2.3.0-6.el8.noarch
sssd-common-2.3.0-6.el8.x86_64
sssd-kcm-2.3.0-6.el8.x86_64
sssd-krb5-2.3.0-6.el8.x86_64
sssd-proxy-2.3.0-6.el8.x86_64
sssd-ipa-2.3.0-6.el8.x86_64
sssd-2.3.0-6.el8.x86_64
sssd-krb5-common-2.3.0-6.el8.x86_64
sssd-ldap-2.3.0-6.el8.x86_64
sssd-common-pac-2.3.0-6.el8.x86_64
sssd-ad-2.3.0-6.el8.x86_64
[root@ci-vm-10-0-138-127 etc]# umask 027
[root@ci-vm-10-0-138-127 etc]# useradd anuj3
[root@ci-vm-10-0-138-127 etc]# vi group(manually add user anuj3 to group wheel)
[root@ci-vm-10-0-138-127 etc]# getent group wheel
wheel:x:10:anuj3
[root@ci-vm-10-0-138-127 etc]# grep wheel group
wheel:x:10:anuj3
[root@ci-vm-10-0-138-127 etc]# pwd
/etc
[root@ci-vm-10-0-138-127 etc]# 
=================================
Comment 22 errata-xmlrpc 2020-11-04 02:05:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sssd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4569