Bug 1008306

+++ This bug was initially created as a clone of Bug #996716 +++

Description of problem:
I Setup a replication server and after reboot it fails to start ipa.  I did the install three different times and every time I'm able to create the replicaton server but after a reboot it fails to start.

After much digging I finally found out that the /var/lock/dirsrv/slapd-INSTANCE dirs are not being created.

/var/lock is a symlink to /run/lock so I updated the tmpfiles.d to point to /run/lock instead and now ipa starts fine on bootup.

Version-Release number of selected component (if applicable):
3.2.2

How reproducible:
Everytime on my VM host when doing replication.
Strange thing is my Master host has the same config files and it starts fine.

Additional Info:

journalctl shows the following failure:
Aug 13 12:45:45 pippin.home.pecknet.com systemd-tmpfiles[191]: Failed to create directory /var/lock/dirsrv: No such file or directory
Aug 13 12:45:45 pippin.home.pecknet.com systemd-tmpfiles[191]: Failed to create directory /var/lock/dirsrv/slapd-HOME-PECKNET-COM: No such file or directory

Could be a race condition where /run/lock doesn't exist yet? (That is created from /usr/lib/tmpfiles.d/legacy)

--- Additional comment from Martin Kosek on 2013-08-14 02:24:19 EDT ---

I noticed the "/var/lock" symlinks in dirsrv's tmpfiles config file:

 # cat /etc/tmpfiles.d/dirsrv-IDM-LAB-BOS-REDHAT-COM.conf 
d /var/run/dirsrv 0770 dirsrv dirsrv
d /var/lock/dirsrv 0770 dirsrv dirsrv
d /var/lock/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM 0770 dirsrv dirsrv

However, is this something that IPA should tackle? It is created during setup-ds.pl phase, by DS. I would expect that DS would update it during upgrade (if necessary). Rich, please advise if this is something that IPA is expected to do.

--- Additional comment from Alexander Bokovoy on 2013-08-14 02:37:37 EDT ---

Similar update to tmpfiles.d needs to be done on Dogtag side as pki-* configs also mention /var/lock and /var/run.

It is worth to file Dogtag bug for that.

As for FreeIPA, we need to avoid running systemd upgrade tool unless we are upgrading from Fedora 16. This needs to be done ASAP.

--- Additional comment from Martin Kosek on 2013-08-14 02:58:06 EDT ---

(In reply to Alexander Bokovoy from comment #2)
> Similar update to tmpfiles.d needs to be done on Dogtag side as pki-*
> configs also mention /var/lock and /var/run.
> 
> It is worth to file Dogtag bug for that.

+1. I cloned this bug.

> As for FreeIPA, we need to avoid running systemd upgrade tool unless we are
> upgrading from Fedora 16. This needs to be done ASAP.

How would you differentiate, that you are upgrading from Fedora 16?

Anyway, I think we should remove this script altogether in both Fedora 19 and Fedora 20. Even Fedora 17 is already end of life, so I do not think we are expected to support upgrades from pre-EOL Fedora to 19 or 20.

--- Additional comment from Nathan Kinder on 2013-08-14 12:43:16 EDT ---

(In reply to Martin Kosek from comment #1)
> I noticed the "/var/lock" symlinks in dirsrv's tmpfiles config file:
> 
>  # cat /etc/tmpfiles.d/dirsrv-IDM-LAB-BOS-REDHAT-COM.conf 
> d /var/run/dirsrv 0770 dirsrv dirsrv
> d /var/lock/dirsrv 0770 dirsrv dirsrv
> d /var/lock/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM 0770 dirsrv dirsrv
> 
> However, is this something that IPA should tackle? It is created during
> setup-ds.pl phase, by DS. I would expect that DS would update it during
> upgrade (if necessary). Rich, please advise if this is something that IPA is
> expected to do.

This does seem to be some sort of race condition, as a fresh install of 389-ds-base on F19 works fine on a reboot in my testing.  We could change this to use /run/lock instead of /var/lock in our tmpfiles.d config in DS, but I don't understand why that should be necessary.  I would like to understand what is happening before making this change.

The fact that using /run/lock instead of /var/lock is working around the  problem indicates that /usr/lib/tmpfiles.d/legacy.conf is being processed before /etc/tmpfiles.d/dirsrv.conf, as the /run/lock directory must already exist.  This doesn't explain why /var/lock wouldn't work, as /var/lock is a persistent symlink on a normal filesystem as far as I can tell.  If /run/lock exists, /var/lock should work just fine.

--- Additional comment from Bill Peck on 2013-08-14 12:57:14 EDT ---

If someone can tell me how to gather more data or enable certain debugging from systemd to try and shed some light on the issue I'd be more than happy to do it.

Should the systemd developers be cc'ed on this?

Thanks

--- Additional comment from Martin Kosek on 2013-08-15 02:24:57 EDT ---

Good idea. Adding Michal of systemd to chime in if this area is familiar to him.

--- Additional comment from Michal Sekletar on 2013-08-21 12:24:45 EDT ---

First of all sorry for not commenting earlier. I think we are hitting an actual bug in systemd-tmpfiles here, since using /run/lock works and /var/lock doesn't. 

Just fyi, tmpfiles drop-in configuration is applied in a way that config files in /etc override those in /usr and after parsing all config files, configuration is then applied according to lexicographical ordering of config file names, but regardless of location.

This means that /etc/tmpfiles.d/dirsrv* config files are applied earlier than /usr/lib/tmpfiles.d/legacy.conf. However, this *shouldn't* matter since we always create whole hierarchy for each path specified in config files and if there is overlap with configuration in later config files then path modes are fixed-up accordingly.

AFAIK, the bug is that that we don't canonicalize path before we start with creating parent hierarchy for path specified in configuration file.

Please use /run paths as a workaround for the time being. Although you might want to keep those and stop using legacy /var paths.

Btw, freeipa-server ships /etc/tmpfiles.d/ipa.conf in package payload. Note that you shouldn't do that as per Fedora Packaging Guidelines policy.

--- Additional comment from Rob Crittenden on 2013-08-27 09:28:58 EDT ---

(In reply to Michal Sekletar from comment #7)
> Btw, freeipa-server ships /etc/tmpfiles.d/ipa.conf in package payload. Note
> that you shouldn't do that as per Fedora Packaging Guidelines policy.

I'm not sure I follow. It looks like we are incorrectly marking it as config, and using the wrong directory, is that what you mean?

--- Additional comment from Michal Sekletar on 2013-08-27 11:31:39 EDT ---

Sorry for confusing statement. I was referring to the path /etc/tmpfiles.d/ipa.conf.

As you correctly pointed out, tmpfiles drop-in shouldn't be in /etc/ and shouldn't be marked as config.

--- Additional comment from Martin Kosek on 2013-09-02 11:25:11 EDT ---

Thanks, I will create an upstream ticket to fix the ipa.conf drop-in and place it in the right directory to avoid violating the guidelines.

--- Additional comment from Martin Kosek on 2013-09-02 11:31:12 EDT ---

Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3894

--- Additional comment from Martin Kosek on 2013-09-03 10:30:01 EDT ---

Upstream ticket in Comment 11 was closed as a duplicate of:
https://fedorahosted.org/freeipa/ticket/3881

--- Additional comment from Orion Poplawski on 2013-09-13 09:36:35 EDT ---

I can edit /etc/tmpfiles.d/dirsrv@NWRA-COM, but then something seems to be changing it back (perhaps updates of 386-ds/ipa).