Bug 2179278

Summary: dirsrv failed to start after reboot because "dirsrv" did not have access on /run/dirsrv
Product: Red Hat Enterprise Linux 9 Reporter: Ding-Yi Chen <dchen>
Component: 389-ds-baseAssignee: LDAP Maintainers <idm-ds-dev-bugs>
Status: VERIFIED --- QA Contact: LDAP QA Team <idm-ds-qe-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 9.1CC: bsmejkal, idm-ds-dev-bugs, mreynolds, vashirov
Target Milestone: rcKeywords: Triaged
Target Release: 9.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: sync-to-jira
Fixed In Version: 389-ds-base-2.3.6-2.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ding-Yi Chen 2023-03-17 07:53:43 UTC 
Description of problem:

After reboot, dirsrv failed to start because /run/dirsrv is not create yet.

~~~
Mar 17 14:58:39 dchen ns-slapd[1994]: [17/Mar/2023:14:58:39.117785245 +1000] - EMERG - main - Unable to access nsslapd-rundir: No such file or directory
Mar 17 14:58:39 dchen ns-slapd[1994]: [17/Mar/2023:14:58:39.127244140 +1000] - EMERG - main - Ensure that user "dirsrv" has read and write permissions on /run/dirsrv
Mar 17 14:58:39 dchen ns-slapd[1994]: [17/Mar/2023:14:58:39.136663531 +1000] - EMERG - main - Shutting down.
Mar 17 14:58:39 dchen journal[2066]: Skipping parental controls support as it’s disabled
Mar 17 14:58:39 dchen systemd[1]: dirsrv: Main process exited, code=exited, status=1/FAILURE
Mar 17 14:58:39 dchen systemd[1]: dirsrv: Failed with result 'exit-code'.
Mar 17 14:58:39 dchen systemd[1]: Failed to start 389 Directory Server EXAMPLE-COM..
Mar 17 14:58:39 dchen ipactl[1686]: Failed to start Directory Service: CalledProcessError(Command ['/bin/systemctl', 'start', 'dirsrv'] returned non-zero exit status 1)
~~~

This is because /run mounted as tmpfs, thus it is possible no dirsrv/ under it
~~~
# mount  | grep /run
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=13107724k,nr_inodes=819200,mode=755,inode64)
~~~


Version-Release number of selected component (if applicable):

389-ds-base-2.1.3-4.el9_1.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Install RH IdM
2. ensure 'nsslapd-rundir: /run/dirsrv' in dse.ldif
3. ensure /run is mounted as tmpfs
4. reboot

Actual results:

Following error message shown:

EMERG - main - Ensure that user "dirsrv" has read and write permissions on /run/dirsrv

Expected results:

389ds create the /run/dirsrv with correct owner and file mod

Additional info:

IPA service that also suffer similar symptom: ipa, named, samba

Workaround:

1. create the directory before 389ds-start, like:
  mkdir -p dirsrv
  chmod -v 770 dirsrv
  chown -v dirsrv:dirsrv dirsrv
  mkdir -p lock/dirsrv/slapd-CHEN-HOME
  chmod -vR 770 lock/dirsrv
  chown -vR dirsrv:dirsrv lock/dirsrv

2. Service systemd-tmpfiles-setup also create the required directories.  

- service ipa, name
Comment 1 Viktor Ashirov 2023-07-14 10:52:21 UTC 
I believe there is a race condition when dirsrv tries to start before systemd-tmpfiles-setup. I couldn't reproduce it myself, but this is not the first time this issue gets reported.

Perhaps we should explicitly require it in our unit file, like so:
After=systemd-tmpfiles-setup.service
Comment 2 Viktor Ashirov 2023-07-26 17:13:40 UTC 
https://github.com/389ds/389-ds-base/pull/5865
Comment 6 Viktor Ashirov 2023-08-14 15:59:32 UTC 
Build tested: 389-ds-base-2.3.6-2.el9.x86_64

# systemctl cat dirsrv@localhost | grep tmpfiles
After=chronyd.service ntpd.service network-online.target systemd-tmpfiles-setup.service
Wants=systemd-tmpfiles-setup.service

# systemctl list-dependencies dirsrv@localhost | grep tmpfiles
● ├─systemd-tmpfiles-setup.service
●   ├─systemd-tmpfiles-setup-dev.service
●   ├─systemd-tmpfiles-setup.service

Marking as Verified:Tested, SanityOnly.