Bug 1412362 - 389-ds-base rpm postinstall script bugs
Summary: 389-ds-base rpm postinstall script bugs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: 389-ds-base
Version: 7.4
Hardware: All
OS: All
unspecified
medium
Target Milestone: rc
: ---
Assignee: mreynolds
QA Contact: Viktor Ashirov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-11 20:53 UTC by Arpit Tolani
Modified: 2020-09-13 21:55 UTC (History)
7 users (show)

Fixed In Version: 389-ds-base-1.3.6.1-3.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 21:14:10 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github 389ds 389-ds-base issues 2147 None None None 2020-09-13 21:55:39 UTC
Red Hat Product Errata RHBA-2017:2086 normal SHIPPED_LIVE 389-ds-base bug fix and enhancement update 2017-08-01 18:37:38 UTC

Description Arpit Tolani 2017-01-11 20:53:08 UTC
Description of problem:

389-ds-base has a postinstall script that intends to add a passwd entry for dirsrv, hunting for an unused uid starting at 389, like this:

dirsrv_uid=389
while [ "`getent passwd | awk -F: '{print $3}' | grep $dirsrv_uid`" != "" ]; do
  dirsrv_uid=`expr $dirsrv_uid + 1`
done

This is bad for several interacting reasons.

First, it downloads the *entire* passwd map for every loop iteration.  This is very expensive for large ldap passwd maps, especially since such downloads are not cached by sssd (or nscd).

Second, the loop termination test is sloppy, because the grep regexp isn't anchored to match the entire uid, so there are many false matches, like this:

shell$ getent passwd | awk -F: '{print $3}' | grep 389
389
11389
23895
23389
5389
3389
10389
23896
23893
23890
12389
23899
23898
22389
23894
23892
23897
23891
24389
13389
25389

It actually loops until it finds a sequence of digits that does not occur in *any* uid, which can require thousands of iterations for large passwd maps.

Third, the combination of many hosts doing many passwd downloads can cause so much ldap abuse that getent passwd occasionally fails, which can cause an already assigned uid to be chosen.

The script similarly tries to update the group map, with similar bad results, but the group termination test is even sloppier ...

while [ "`getent group | grep $dirsrv_gid`" != "" ]; do

... which incorrectly matches the proposed gid *anywhere* within each group line.

Version-Release number of selected component (if applicable):
389-ds-base-1.3.5.10-12.el7_3.x86_64

How reproducible:
Always

Comment 2 Rich Megginson 2017-01-11 22:05:59 UTC
Do you need to know what the uid and gid values are?  From the useradd man page:

       -r, --system
           Create a system account.

           System users will be created with no aging information in
           /etc/shadow, and their numeric identifiers are chosen in the
           SYS_UID_MIN-SYS_UID_MAX range, defined in /etc/login.defs, instead

       -u, --uid UID
           The numerical value of the user's ID. This value must be unique,
           unless the -o option is used. The value must be non-negative. The
           default is to use the smallest ID value greater than or equal to
           UID_MIN and greater than every other user.

So, presumably, you could just check if 389 is in use.  If not, call useradd with -u 389.  If so, call useradd without -u, and let the system generate the number.  I'm assuming groupadd would work the same way.

Comment 4 Noriko Hosoi 2017-01-12 03:19:12 UTC
Upstream ticket:
https://fedorahosted.org/389/ticket/49088

Comment 8 Arpit Tolani 2017-01-12 18:07:10 UTC
(In reply to Rich Megginson from comment #2)

> So, presumably, you could just check if 389 is in use.  If not, call useradd
> with -u 389.  If so, call useradd without -u, and let the system generate
> the number.  I'm assuming groupadd would work the same way.


Do you mean, 

 If not, call useradd with -u 389.  If so, call useradd without -u & with -r, and let the system generate the number from range of system users.

Comment 10 Rich Megginson 2017-01-12 18:17:34 UTC
(In reply to Arpit Tolani from comment #8)
> (In reply to Rich Megginson from comment #2)
> 
> > So, presumably, you could just check if 389 is in use.  If not, call useradd
> > with -u 389.  If so, call useradd without -u, and let the system generate
> > the number.  I'm assuming groupadd would work the same way.
> 
> 
> Do you mean, 
> 
>  If not, call useradd with -u 389.  If so, call useradd without -u & with
> -r, and let the system generate the number from range of system users.

Correct.

Comment 12 Frank Hirtz 2017-01-13 21:21:43 UTC
Is there a reason why we might not want to use the upstream template to do this rather than a novel implementation?

https://fedoraproject.org/wiki/Packaging:UsersAndGroups

From that, we have something along the following lines:

<snip>
To create users and groups in packages, use the following:
Requires(pre): shadow-utils
[...] 
%pre
getent group GROUPNAME >/dev/null || groupadd -f -g ALLOCATED_GID -r GROUPNAME
if ! getent passwd USERNAME >/dev/null ; then
    if ! getent passwd ALLOCATED_UID >/dev/null ; then
      useradd -r -u ALLOCATED_UID -g GROUPNAME -d HOMEDIR -s /sbin/nologin -c "Useful comment about the purpose of this account" USERNAME
    else
      useradd -r -g GROUPNAME -d HOMEDIR -s /sbin/nologin -c "Useful comment about the purpose of this account" USERNAME
    fi
fi
exit 0
</snip>

Comment 19 Sankar Ramalingam 2017-05-16 02:13:06 UTC
[0 root@qeos-43 ~]# getent passwd | awk -F: '{print $3}' | grep 389
389

getent passwd lists only 389. Should this be good enough to verify this bug?

Comment 20 Arpit Tolani 2017-05-16 20:53:51 UTC
(In reply to Sankar Ramalingam from comment #19)
> [0 root@qeos-43 ~]# getent passwd | awk -F: '{print $3}' | grep 389
> 389
> 
> getent passwd lists only 389. Should this be good enough to verify this bug?

No, It will list other users also if their uid contains 389.

Example

# getent passwd system
system:x:1389:1389::/home/system:

# getent passwd | awk -F: '{print $3}' | grep 389
389
1389

Comment 21 Sankar Ramalingam 2017-06-14 15:13:14 UTC
Hi Arpit,
    I understand this is an internal change in the post install script of 389-ds-base rpm, but not clear with the exact steps to verify. I read through the bug and the patch in this ticket - https://pagure.io/389-ds-base/issue/49088, but no clear instructions to verify the bug. 

If I understand correctly, then I need to add some bogus entries with "*389" in /etc/passwd and /etc/group and proceed with installation of 389-ds-base and check if it correctly adds dirsrv entries in /etc/passwd and /etc/group...

In which case, you expect the test to FAIL and what will be the failure?

Comment 22 Sankar Ramalingam 2017-06-15 12:50:41 UTC
I created few users with "*389*" as uid/gid and then using "ipa user-add" command added an entry and duplicate entry to check the time taken.

[root@intel-chiefriver-02 ~]# getent passwd | awk -F: '{print $3}' | grep 389 |wc -l
39
[root@intel-chiefriver-02 ~]# time ipa user-add sramling  --first=Sankar --last Ramalingam
---------------------
Added user "sramling"
---------------------
  User login: sramling
  First name: Sankar
  Last name: Ramalingam
  Full name: Sankar Ramalingam
  Display name: Sankar Ramalingam
  Initials: SR
  Home directory: /home/sramling
  GECOS: Sankar Ramalingam
  Login shell: /bin/sh
  Principal name: sramling@KHW.LAB.ENG.BOS.REDHAT.COM
  Principal alias: sramling@KHW.LAB.ENG.BOS.REDHAT.COM
  Email address: sramling@khw.lab.eng.bos.redhat.com
  UID: 1162200004
  GID: 1162200004
  Password: False
  Member of groups: ipausers
  Kerberos keys available: False

real	0m0.628s
user	0m0.479s
sys	0m0.033s
[root@intel-chiefriver-02 ~]# time ipa user-add sramling  --first=Sankar --last Ramalingam
ipa: ERROR: user with name "sramling" already exists

real	0m1.374s
user	0m0.472s
sys	0m0.036s
[root@intel-chiefriver-02 ~]# time ipa user-add sramling  --first=Sankar --last Ramalingam
ipa: ERROR: user with name "sramling" already exists

real	0m0.913s
user	0m0.472s
sys	0m0.037s
[root@intel-chiefriver-02 ~]# time ipa user-add sramling  --first=Sankar --last Ramalingam
ipa: ERROR: user with name "sramling" already exists

real	0m0.577s
user	0m0.461s
sys	0m0.047s
[root@intel-chiefriver-02 ~]# time ipa user-add sramling  --first=Sankar --last Ramalingam
ipa: ERROR: user with name "sramling" already exists

real	0m0.571s
user	0m0.473s
sys	0m0.037s
[root@intel-chiefriver-02 ~]# time ipa user-add sramling  --first=Sankar --last Ramalingam
ipa: ERROR: user with name "sramling" already exists

real	0m0.583s
user	0m0.459s
sys	0m0.049s

It looks like when the user doesn't exist, then it completes the user-add operation in less than 1 sec. But, when I tried adding the same user, it took about 1.374 secs. The consecutive add took less than 0.6 secs. That means, the genent was cached and it took less time to complete the operation. Correct me if I am wrong.

I tested the same above test case again, but, there is no delay observed.

[root@intel-chiefriver-02 ~]# time ipa user-add Bsramling  --first=BSankar --last BRamalingam
----------------------
Added user "bsramling"
----------------------
  User login: bsramling
  First name: BSankar
  Last name: BRamalingam
  Full name: BSankar BRamalingam
  Display name: BSankar BRamalingam
  Initials: BB
  Home directory: /home/bsramling
  GECOS: BSankar BRamalingam
  Login shell: /bin/sh
  Principal name: bsramling@KHW.LAB.ENG.BOS.REDHAT.COM
  Principal alias: bsramling@KHW.LAB.ENG.BOS.REDHAT.COM
  Email address: bsramling@khw.lab.eng.bos.redhat.com
  UID: 1162200008
  GID: 1162200008
  Password: False
  Member of groups: ipausers
  Kerberos keys available: False

real	0m0.591s
user	0m0.467s
sys	0m0.045s
[root@intel-chiefriver-02 ~]# time ipa user-add Bsramling  --first=BSankar --last BRamalingam
ipa: ERROR: user with name "bsramling" already exists

real	0m0.549s
user	0m0.479s
sys	0m0.028s

Comment 23 Sankar Ramalingam 2017-06-15 16:19:39 UTC
Installed IPA server and created multiple entries containing "389" in uid and gid. Then, compared the performance between 389-ds-base-1.3.5.10-21 and 389-ds-base-1.3.6.1-16. The installation was faster on 1.3.6.1-16. Hence, marking the bug as Verified

[root@vm-idm-007 ~]# ldapsearch -p 389 -h localhost -D "cn=Directory manager" -w Secret123 -b "dc=testrelm,dc=test" "objectClass=*" |egrep 'uid|gid' |grep  389 |wc -l
156


[root@vm-idm-001 ~]# rpm -q --scripts 389-ds-base |grep -i getent
getent group $GROUPNAME >/dev/null || /usr/sbin/groupadd -f -g $ALLOCATED_GID -r $GROUPNAME
if ! getent passwd $USERNAME >/dev/null ; then
    if ! getent passwd $ALLOCATED_UID >/dev/null ; then
[root@vm-idm-001 ~]# rpm -qa |grep -i 389-ds-base
389-ds-base-libs-1.3.6.1-16.el7.x86_64
389-ds-base-1.3.6.1-16.el7.x86_64

Complete!

real	0m8.509s
user	0m6.470s
sys	0m0.781s


[root@srclient ~]# rpm -q --scripts 389-ds-base |grep -i getent
  while [ "`getent passwd | awk -F: '{print $3}' | grep $dirsrv_uid`" != "" ]; do
  while [ "`getent group | grep $dirsrv_gid`" != "" ]; do
[root@srclient ~]# rpm -qa |grep -i 389-ds
389-ds-base-libs-1.3.5.10-21.el7_3.x86_64
389-ds-base-1.3.5.10-21.el7_3.x86_64

Complete!

real	0m15.196s
user	0m6.828s
sys	0m0.844s

Comment 24 Arpit Tolani 2017-06-23 15:08:56 UTC
(In reply to Sankar Ramalingam from comment #21) 

This bug has nothing to do with configuring IPA & adding user after IPA installation. 

Issue was while installing IPA server or RHDS packages, To check if dirsrv user is already added or not, It was trying to search entire passwd tree which was very sloppy.  It has been fixed. 

24 + # Soft static allocation for UID and GID
25 + USERNAME="dirsrv"
26 + ALLOCATED_UID=389
27 + GROUPNAME="dirsrv"
28 + ALLOCATED_GID=389
29 + HOMEDIR="/usr/share/dirsrv"
30 + 
31 + getent group $GROUPNAME >/dev/null || groupadd -f -g $ALLOCATED_GID -r $GROUPNAME
32 + if ! getent passwd $USERNAME >/dev/null ; then
33 +     if ! getent passwd $ALLOCATED_UID >/dev/null ; then
34 +       useradd -r -u $ALLOCATED_UID -g $GROUPNAME -d $HOMEDIR -s /sbin/nologin -c "user for 389-ds-base" $USERNAME
35 +     else
36 +       useradd -r -g $GROUPNAME -d $HOMEDIR -s /sbin/nologin -c "user for 389-ds-base" $USERNAME
37 +     fi

I don't see a very good way to verify it apart from running it in bash debug & checking if its trying to download the entire tree while installing packages or not.

Or best is marking it verified sanity only.

Comment 25 errata-xmlrpc 2017-08-01 21:14:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2086


Note You need to log in before you can comment on or make changes to this bug.