+++ This bug was initially created as a clone of Bug #996716 +++ Description of problem: I Setup a replication server and after reboot it fails to start ipa. I did the install three different times and every time I'm able to create the replicaton server but after a reboot it fails to start. After much digging I finally found out that the /var/lock/dirsrv/slapd-INSTANCE dirs are not being created. /var/lock is a symlink to /run/lock so I updated the tmpfiles.d to point to /run/lock instead and now ipa starts fine on bootup. Version-Release number of selected component (if applicable): 3.2.2 How reproducible: Everytime on my VM host when doing replication. Strange thing is my Master host has the same config files and it starts fine. Additional Info: journalctl shows the following failure: Aug 13 12:45:45 pippin.home.pecknet.com systemd-tmpfiles[191]: Failed to create directory /var/lock/dirsrv: No such file or directory Aug 13 12:45:45 pippin.home.pecknet.com systemd-tmpfiles[191]: Failed to create directory /var/lock/dirsrv/slapd-HOME-PECKNET-COM: No such file or directory Could be a race condition where /run/lock doesn't exist yet? (That is created from /usr/lib/tmpfiles.d/legacy) --- Additional comment from Martin Kosek on 2013-08-14 02:24:19 EDT --- I noticed the "/var/lock" symlinks in dirsrv's tmpfiles config file: # cat /etc/tmpfiles.d/dirsrv-IDM-LAB-BOS-REDHAT-COM.conf d /var/run/dirsrv 0770 dirsrv dirsrv d /var/lock/dirsrv 0770 dirsrv dirsrv d /var/lock/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM 0770 dirsrv dirsrv However, is this something that IPA should tackle? It is created during setup-ds.pl phase, by DS. I would expect that DS would update it during upgrade (if necessary). Rich, please advise if this is something that IPA is expected to do. --- Additional comment from Alexander Bokovoy on 2013-08-14 02:37:37 EDT --- Similar update to tmpfiles.d needs to be done on Dogtag side as pki-* configs also mention /var/lock and /var/run. It is worth to file Dogtag bug for that. As for FreeIPA, we need to avoid running systemd upgrade tool unless we are upgrading from Fedora 16. This needs to be done ASAP. --- Additional comment from Martin Kosek on 2013-08-14 02:58:06 EDT --- (In reply to Alexander Bokovoy from comment #2) > Similar update to tmpfiles.d needs to be done on Dogtag side as pki-* > configs also mention /var/lock and /var/run. > > It is worth to file Dogtag bug for that. +1. I cloned this bug. > As for FreeIPA, we need to avoid running systemd upgrade tool unless we are > upgrading from Fedora 16. This needs to be done ASAP. How would you differentiate, that you are upgrading from Fedora 16? Anyway, I think we should remove this script altogether in both Fedora 19 and Fedora 20. Even Fedora 17 is already end of life, so I do not think we are expected to support upgrades from pre-EOL Fedora to 19 or 20. --- Additional comment from Nathan Kinder on 2013-08-14 12:43:16 EDT --- (In reply to Martin Kosek from comment #1) > I noticed the "/var/lock" symlinks in dirsrv's tmpfiles config file: > > # cat /etc/tmpfiles.d/dirsrv-IDM-LAB-BOS-REDHAT-COM.conf > d /var/run/dirsrv 0770 dirsrv dirsrv > d /var/lock/dirsrv 0770 dirsrv dirsrv > d /var/lock/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM 0770 dirsrv dirsrv > > However, is this something that IPA should tackle? It is created during > setup-ds.pl phase, by DS. I would expect that DS would update it during > upgrade (if necessary). Rich, please advise if this is something that IPA is > expected to do. This does seem to be some sort of race condition, as a fresh install of 389-ds-base on F19 works fine on a reboot in my testing. We could change this to use /run/lock instead of /var/lock in our tmpfiles.d config in DS, but I don't understand why that should be necessary. I would like to understand what is happening before making this change. The fact that using /run/lock instead of /var/lock is working around the problem indicates that /usr/lib/tmpfiles.d/legacy.conf is being processed before /etc/tmpfiles.d/dirsrv.conf, as the /run/lock directory must already exist. This doesn't explain why /var/lock wouldn't work, as /var/lock is a persistent symlink on a normal filesystem as far as I can tell. If /run/lock exists, /var/lock should work just fine. --- Additional comment from Bill Peck on 2013-08-14 12:57:14 EDT --- If someone can tell me how to gather more data or enable certain debugging from systemd to try and shed some light on the issue I'd be more than happy to do it. Should the systemd developers be cc'ed on this? Thanks --- Additional comment from Martin Kosek on 2013-08-15 02:24:57 EDT --- Good idea. Adding Michal of systemd to chime in if this area is familiar to him. --- Additional comment from Michal Sekletar on 2013-08-21 12:24:45 EDT --- First of all sorry for not commenting earlier. I think we are hitting an actual bug in systemd-tmpfiles here, since using /run/lock works and /var/lock doesn't. Just fyi, tmpfiles drop-in configuration is applied in a way that config files in /etc override those in /usr and after parsing all config files, configuration is then applied according to lexicographical ordering of config file names, but regardless of location. This means that /etc/tmpfiles.d/dirsrv* config files are applied earlier than /usr/lib/tmpfiles.d/legacy.conf. However, this *shouldn't* matter since we always create whole hierarchy for each path specified in config files and if there is overlap with configuration in later config files then path modes are fixed-up accordingly. AFAIK, the bug is that that we don't canonicalize path before we start with creating parent hierarchy for path specified in configuration file. Please use /run paths as a workaround for the time being. Although you might want to keep those and stop using legacy /var paths. Btw, freeipa-server ships /etc/tmpfiles.d/ipa.conf in package payload. Note that you shouldn't do that as per Fedora Packaging Guidelines policy. --- Additional comment from Rob Crittenden on 2013-08-27 09:28:58 EDT --- (In reply to Michal Sekletar from comment #7) > Btw, freeipa-server ships /etc/tmpfiles.d/ipa.conf in package payload. Note > that you shouldn't do that as per Fedora Packaging Guidelines policy. I'm not sure I follow. It looks like we are incorrectly marking it as config, and using the wrong directory, is that what you mean? --- Additional comment from Michal Sekletar on 2013-08-27 11:31:39 EDT --- Sorry for confusing statement. I was referring to the path /etc/tmpfiles.d/ipa.conf. As you correctly pointed out, tmpfiles drop-in shouldn't be in /etc/ and shouldn't be marked as config. --- Additional comment from Martin Kosek on 2013-09-02 11:25:11 EDT --- Thanks, I will create an upstream ticket to fix the ipa.conf drop-in and place it in the right directory to avoid violating the guidelines. --- Additional comment from Martin Kosek on 2013-09-02 11:31:12 EDT --- Upstream ticket: https://fedorahosted.org/freeipa/ticket/3894 --- Additional comment from Martin Kosek on 2013-09-03 10:30:01 EDT --- Upstream ticket in Comment 11 was closed as a duplicate of: https://fedorahosted.org/freeipa/ticket/3881 --- Additional comment from Orion Poplawski on 2013-09-13 09:36:35 EDT --- I can edit /etc/tmpfiles.d/dirsrv@NWRA-COM, but then something seems to be changing it back (perhaps updates of 386-ds/ipa).
Upstream ticket: https://fedorahosted.org/389/ticket/47513
This has been fixed in the 389-ds-base source repo. The fix will be available in 389-ds-base-1.3.1.10.
Just to verify, that build will make F19, right, it's not a dev-builds-only release? As this is a fairly major bug.
(In reply to Adam Williamson from comment #3) > Just to verify, that build will make F19, right, it's not a dev-builds-only > release? As this is a fairly major bug. Yes, it will be built for inclusion in F19 and later. I believe the plan is to build it today. Once it is built, it would be great if you could test it and provide karma in Bodhi to help get the package moved to the stable repo.
389-ds-base-1.3.1.10-1.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/389-ds-base-1.3.1.10-1.fc19
389-ds-base-1.3.1.10-1.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.
(In reply to Fedora Update System from comment #6) > 389-ds-base-1.3.1.10-1.fc19 has been pushed to the Fedora 19 stable > repository. If problems still persist, please make note of it in this bug > report. I wish I could have caught this sooner, but the update to 1.3.1.10-1.fc19 causes some breakage. Prior to the update, my /etc/tmpfiles.d/dirsrv-EXAMPLE-COM.conf (FreeIPA) contained: tmpfiles.d]# cat dirsrv-EXAMPLE-COM.conf d /var/run/dirsrv 0770 dirsrv dirsrv d /var/lock/dirsrv 0770 dirsrv dirsrv d /var/lock/dirsrv/slapd-EXAMPLE-COM 0770 dirsrv dirsrv After the update, it contains: tmpfiles.d]# cat dirsrv-EXAMPLE-COM.conf d /var/dirsrv 0770 dirsrv dirsrv d /var/lock/dirsrv 0770 dirsrv dirsrv d /var/lock/dirsrv/slapd-EXAMPLE-COM 0770 dirsrv dirsrv and causes an SELinux denial, as /var/dirsrv can't be created at bootup. For now, to resolve the issue, I've changed d /var/dirsrv 0770 dirsrv dirsrv to d /var/run/dirsrv 0770 dirsrv dirsrv
Yes, there is something wrong with 1.3.1-10. I confirmed that the source in the 1.3.1 branch is correct and works, but 1.3.1-10 is broken. 1.3.1-10 should have these changes, so I'm not sure what went wrong.
389-ds-base-1.3.1.11-1.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/389-ds-base-1.3.1.11-1.fc19
Sorry I wasn't able to test and catch this; it came at a bad time as I was travelling between continents. As I'm now several thousand miles away from the place where the physical machines live, I'm not keen on fiddling with the working setup too much, as if I happen to knock it all sideways, it could be tricky/impossible to fix :/
389-ds-base-1.3.1.11-1.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.
389-ds-base-1.3.1.12-1.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/389-ds-base-1.3.1.12-1.fc19
389-ds-base-1.3.2.2-1.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/389-ds-base-1.3.2.2-1.fc20
Package 389-ds-base-1.3.1.12-1.fc19: * should fix your issue, * was pushed to the Fedora 19 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing 389-ds-base-1.3.1.12-1.fc19' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-18914/389-ds-base-1.3.1.12-1.fc19 then log in and leave karma (feedback).
389-ds-base-1.3.2.2-1.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
389-ds-base-1.3.1.12-1.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.