Bug 1008306 - tmpfiles.d references /var/lock when they should reference /run/lock
tmpfiles.d references /var/lock when they should reference /run/lock
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: 389-ds-base (Show other bugs)
19
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Rich Megginson
Fedora Extras Quality Assurance
:
Depends On: 996716
Blocks: 996847 1008610
  Show dependency treegraph
 
Reported: 2013-09-16 02:18 EDT by Martin Kosek
Modified: 2013-10-22 01:03 EDT (History)
16 users (show)

See Also:
Fixed In Version: 389-ds-base-1.3.1.12-1.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 996716
: 1008610 (view as bug list)
Environment:
Last Closed: 2013-10-15 02:31:43 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Martin Kosek 2013-09-16 02:18:05 EDT
+++ This bug was initially created as a clone of Bug #996716 +++

Description of problem:
I Setup a replication server and after reboot it fails to start ipa.  I did the install three different times and every time I'm able to create the replicaton server but after a reboot it fails to start.

After much digging I finally found out that the /var/lock/dirsrv/slapd-INSTANCE dirs are not being created.

/var/lock is a symlink to /run/lock so I updated the tmpfiles.d to point to /run/lock instead and now ipa starts fine on bootup.

Version-Release number of selected component (if applicable):
3.2.2

How reproducible:
Everytime on my VM host when doing replication.
Strange thing is my Master host has the same config files and it starts fine.

Additional Info:

journalctl shows the following failure:
Aug 13 12:45:45 pippin.home.pecknet.com systemd-tmpfiles[191]: Failed to create directory /var/lock/dirsrv: No such file or directory
Aug 13 12:45:45 pippin.home.pecknet.com systemd-tmpfiles[191]: Failed to create directory /var/lock/dirsrv/slapd-HOME-PECKNET-COM: No such file or directory

Could be a race condition where /run/lock doesn't exist yet? (That is created from /usr/lib/tmpfiles.d/legacy)

--- Additional comment from Martin Kosek on 2013-08-14 02:24:19 EDT ---

I noticed the "/var/lock" symlinks in dirsrv's tmpfiles config file:

 # cat /etc/tmpfiles.d/dirsrv-IDM-LAB-BOS-REDHAT-COM.conf 
d /var/run/dirsrv 0770 dirsrv dirsrv
d /var/lock/dirsrv 0770 dirsrv dirsrv
d /var/lock/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM 0770 dirsrv dirsrv

However, is this something that IPA should tackle? It is created during setup-ds.pl phase, by DS. I would expect that DS would update it during upgrade (if necessary). Rich, please advise if this is something that IPA is expected to do.

--- Additional comment from Alexander Bokovoy on 2013-08-14 02:37:37 EDT ---

Similar update to tmpfiles.d needs to be done on Dogtag side as pki-* configs also mention /var/lock and /var/run.

It is worth to file Dogtag bug for that.

As for FreeIPA, we need to avoid running systemd upgrade tool unless we are upgrading from Fedora 16. This needs to be done ASAP.

--- Additional comment from Martin Kosek on 2013-08-14 02:58:06 EDT ---

(In reply to Alexander Bokovoy from comment #2)
> Similar update to tmpfiles.d needs to be done on Dogtag side as pki-*
> configs also mention /var/lock and /var/run.
> 
> It is worth to file Dogtag bug for that.

+1. I cloned this bug.

> As for FreeIPA, we need to avoid running systemd upgrade tool unless we are
> upgrading from Fedora 16. This needs to be done ASAP.

How would you differentiate, that you are upgrading from Fedora 16?

Anyway, I think we should remove this script altogether in both Fedora 19 and Fedora 20. Even Fedora 17 is already end of life, so I do not think we are expected to support upgrades from pre-EOL Fedora to 19 or 20.

--- Additional comment from Nathan Kinder on 2013-08-14 12:43:16 EDT ---

(In reply to Martin Kosek from comment #1)
> I noticed the "/var/lock" symlinks in dirsrv's tmpfiles config file:
> 
>  # cat /etc/tmpfiles.d/dirsrv-IDM-LAB-BOS-REDHAT-COM.conf 
> d /var/run/dirsrv 0770 dirsrv dirsrv
> d /var/lock/dirsrv 0770 dirsrv dirsrv
> d /var/lock/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM 0770 dirsrv dirsrv
> 
> However, is this something that IPA should tackle? It is created during
> setup-ds.pl phase, by DS. I would expect that DS would update it during
> upgrade (if necessary). Rich, please advise if this is something that IPA is
> expected to do.

This does seem to be some sort of race condition, as a fresh install of 389-ds-base on F19 works fine on a reboot in my testing.  We could change this to use /run/lock instead of /var/lock in our tmpfiles.d config in DS, but I don't understand why that should be necessary.  I would like to understand what is happening before making this change.

The fact that using /run/lock instead of /var/lock is working around the  problem indicates that /usr/lib/tmpfiles.d/legacy.conf is being processed before /etc/tmpfiles.d/dirsrv.conf, as the /run/lock directory must already exist.  This doesn't explain why /var/lock wouldn't work, as /var/lock is a persistent symlink on a normal filesystem as far as I can tell.  If /run/lock exists, /var/lock should work just fine.

--- Additional comment from Bill Peck on 2013-08-14 12:57:14 EDT ---

If someone can tell me how to gather more data or enable certain debugging from systemd to try and shed some light on the issue I'd be more than happy to do it.

Should the systemd developers be cc'ed on this?

Thanks

--- Additional comment from Martin Kosek on 2013-08-15 02:24:57 EDT ---

Good idea. Adding Michal of systemd to chime in if this area is familiar to him.

--- Additional comment from Michal Sekletar on 2013-08-21 12:24:45 EDT ---

First of all sorry for not commenting earlier. I think we are hitting an actual bug in systemd-tmpfiles here, since using /run/lock works and /var/lock doesn't. 

Just fyi, tmpfiles drop-in configuration is applied in a way that config files in /etc override those in /usr and after parsing all config files, configuration is then applied according to lexicographical ordering of config file names, but regardless of location.

This means that /etc/tmpfiles.d/dirsrv* config files are applied earlier than /usr/lib/tmpfiles.d/legacy.conf. However, this *shouldn't* matter since we always create whole hierarchy for each path specified in config files and if there is overlap with configuration in later config files then path modes are fixed-up accordingly.

AFAIK, the bug is that that we don't canonicalize path before we start with creating parent hierarchy for path specified in configuration file.

Please use /run paths as a workaround for the time being. Although you might want to keep those and stop using legacy /var paths.

Btw, freeipa-server ships /etc/tmpfiles.d/ipa.conf in package payload. Note that you shouldn't do that as per Fedora Packaging Guidelines policy.

--- Additional comment from Rob Crittenden on 2013-08-27 09:28:58 EDT ---

(In reply to Michal Sekletar from comment #7)
> Btw, freeipa-server ships /etc/tmpfiles.d/ipa.conf in package payload. Note
> that you shouldn't do that as per Fedora Packaging Guidelines policy.

I'm not sure I follow. It looks like we are incorrectly marking it as config, and using the wrong directory, is that what you mean?

--- Additional comment from Michal Sekletar on 2013-08-27 11:31:39 EDT ---

Sorry for confusing statement. I was referring to the path /etc/tmpfiles.d/ipa.conf.

As you correctly pointed out, tmpfiles drop-in shouldn't be in /etc/ and shouldn't be marked as config.

--- Additional comment from Martin Kosek on 2013-09-02 11:25:11 EDT ---

Thanks, I will create an upstream ticket to fix the ipa.conf drop-in and place it in the right directory to avoid violating the guidelines.

--- Additional comment from Martin Kosek on 2013-09-02 11:31:12 EDT ---

Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3894

--- Additional comment from Martin Kosek on 2013-09-03 10:30:01 EDT ---

Upstream ticket in Comment 11 was closed as a duplicate of:
https://fedorahosted.org/freeipa/ticket/3881

--- Additional comment from Orion Poplawski on 2013-09-13 09:36:35 EDT ---

I can edit /etc/tmpfiles.d/dirsrv@NWRA-COM, but then something seems to be changing it back (perhaps updates of 386-ds/ipa).
Comment 1 Nathan Kinder 2013-09-16 11:11:08 EDT
Upstream ticket:
https://fedorahosted.org/389/ticket/47513
Comment 2 Nathan Kinder 2013-09-27 11:48:30 EDT
This has been fixed in the 389-ds-base source repo.  The fix will be available in 389-ds-base-1.3.1.10.
Comment 3 Adam Williamson 2013-09-27 12:28:47 EDT
Just to verify, that build will make F19, right, it's not a dev-builds-only release? As this is a fairly major bug.
Comment 4 Nathan Kinder 2013-09-27 12:42:13 EDT
(In reply to Adam Williamson from comment #3)
> Just to verify, that build will make F19, right, it's not a dev-builds-only
> release? As this is a fairly major bug.

Yes, it will be built for inclusion in F19 and later.  I believe the plan is to build it today.  Once it is built, it would be great if you could test it and provide karma in Bodhi to help get the package moved to the stable repo.
Comment 5 Fedora Update System 2013-09-27 17:15:57 EDT
389-ds-base-1.3.1.10-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/389-ds-base-1.3.1.10-1.fc19
Comment 6 Fedora Update System 2013-09-29 20:46:02 EDT
389-ds-base-1.3.1.10-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 7 Anthony Messina 2013-09-30 09:55:40 EDT
(In reply to Fedora Update System from comment #6)
> 389-ds-base-1.3.1.10-1.fc19 has been pushed to the Fedora 19 stable
> repository.  If problems still persist, please make note of it in this bug
> report.

I wish I could have caught this sooner, but the update to 1.3.1.10-1.fc19 causes some breakage.  Prior to the update, my /etc/tmpfiles.d/dirsrv-EXAMPLE-COM.conf (FreeIPA) contained:

tmpfiles.d]# cat dirsrv-EXAMPLE-COM.conf 
d /var/run/dirsrv 0770 dirsrv dirsrv
d /var/lock/dirsrv 0770 dirsrv dirsrv
d /var/lock/dirsrv/slapd-EXAMPLE-COM 0770 dirsrv dirsrv

After the update, it contains:

tmpfiles.d]# cat dirsrv-EXAMPLE-COM.conf 
d /var/dirsrv 0770 dirsrv dirsrv
d /var/lock/dirsrv 0770 dirsrv dirsrv
d /var/lock/dirsrv/slapd-EXAMPLE-COM 0770 dirsrv dirsrv

and causes an SELinux denial, as /var/dirsrv can't be created at bootup.  For now, to resolve the issue, I've changed

d /var/dirsrv 0770 dirsrv dirsrv

to

d /var/run/dirsrv 0770 dirsrv dirsrv
Comment 8 mreynolds 2013-09-30 10:46:10 EDT
Yes, there is something wrong with 1.3.1-10.

I confirmed that the source in the 1.3.1 branch is correct and works, but 1.3.1-10 is broken.  1.3.1-10 should have these changes, so I'm not sure what went wrong.
Comment 9 Fedora Update System 2013-09-30 14:25:30 EDT
389-ds-base-1.3.1.11-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/389-ds-base-1.3.1.11-1.fc19
Comment 10 Adam Williamson 2013-10-01 08:50:48 EDT
Sorry I wasn't able to test and catch this; it came at a bad time as I was travelling between continents. As I'm now several thousand miles away from the place where the physical machines live, I'm not keen on fiddling with the working setup too much, as if I happen to knock it all sideways, it could be tricky/impossible to fix :/
Comment 11 Fedora Update System 2013-10-08 07:24:31 EDT
389-ds-base-1.3.1.11-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 12 Fedora Update System 2013-10-10 20:42:53 EDT
389-ds-base-1.3.1.12-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/389-ds-base-1.3.1.12-1.fc19
Comment 13 Fedora Update System 2013-10-11 15:22:08 EDT
389-ds-base-1.3.2.2-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/389-ds-base-1.3.2.2-1.fc20
Comment 14 Fedora Update System 2013-10-11 19:59:34 EDT
Package 389-ds-base-1.3.1.12-1.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing 389-ds-base-1.3.1.12-1.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-18914/389-ds-base-1.3.1.12-1.fc19
then log in and leave karma (feedback).
Comment 15 Fedora Update System 2013-10-15 02:31:43 EDT
389-ds-base-1.3.2.2-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 16 Fedora Update System 2013-10-22 01:03:55 EDT
389-ds-base-1.3.1.12-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.