Bug 996716 - tmpfiles.d references /var/lock when they should reference /run/lock
tmpfiles.d references /var/lock when they should reference /run/lock
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: freeipa (Show other bugs)
19
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Rob Crittenden
Fedora Extras Quality Assurance
:
: 983073 (view as bug list)
Depends On:
Blocks: 996847 1008306 1008610
  Show dependency treegraph
 
Reported: 2013-08-13 15:13 EDT by Bill Peck
Modified: 2013-10-14 13:09 EDT (History)
12 users (show)

See Also:
Fixed In Version: freeipa-3.3.2-1.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 996847 1008306 (view as bug list)
Environment:
Last Closed: 2013-10-14 02:58:23 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Bill Peck 2013-08-13 15:13:16 EDT
Description of problem:
I Setup a replication server and after reboot it fails to start ipa.  I did the install three different times and every time I'm able to create the replicaton server but after a reboot it fails to start.

After much digging I finally found out that the /var/lock/dirsrv/slapd-INSTANCE dirs are not being created.

/var/lock is a symlink to /run/lock so I updated the tmpfiles.d to point to /run/lock instead and now ipa starts fine on bootup.

Version-Release number of selected component (if applicable):
3.2.2

How reproducible:
Everytime on my VM host when doing replication.
Strange thing is my Master host has the same config files and it starts fine.

Additional Info:

journalctl shows the following failure:
Aug 13 12:45:45 pippin.home.pecknet.com systemd-tmpfiles[191]: Failed to create directory /var/lock/dirsrv: No such file or directory
Aug 13 12:45:45 pippin.home.pecknet.com systemd-tmpfiles[191]: Failed to create directory /var/lock/dirsrv/slapd-HOME-PECKNET-COM: No such file or directory

Could be a race condition where /run/lock doesn't exist yet? (That is created from /usr/lib/tmpfiles.d/legacy)
Comment 1 Martin Kosek 2013-08-14 02:24:19 EDT
I noticed the "/var/lock" symlinks in dirsrv's tmpfiles config file:

 # cat /etc/tmpfiles.d/dirsrv-IDM-LAB-BOS-REDHAT-COM.conf 
d /var/run/dirsrv 0770 dirsrv dirsrv
d /var/lock/dirsrv 0770 dirsrv dirsrv
d /var/lock/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM 0770 dirsrv dirsrv

However, is this something that IPA should tackle? It is created during setup-ds.pl phase, by DS. I would expect that DS would update it during upgrade (if necessary). Rich, please advise if this is something that IPA is expected to do.
Comment 2 Alexander Bokovoy 2013-08-14 02:37:37 EDT
Similar update to tmpfiles.d needs to be done on Dogtag side as pki-* configs also mention /var/lock and /var/run.

It is worth to file Dogtag bug for that.

As for FreeIPA, we need to avoid running systemd upgrade tool unless we are upgrading from Fedora 16. This needs to be done ASAP.
Comment 3 Martin Kosek 2013-08-14 02:58:06 EDT
(In reply to Alexander Bokovoy from comment #2)
> Similar update to tmpfiles.d needs to be done on Dogtag side as pki-*
> configs also mention /var/lock and /var/run.
> 
> It is worth to file Dogtag bug for that.

+1. I cloned this bug.

> As for FreeIPA, we need to avoid running systemd upgrade tool unless we are
> upgrading from Fedora 16. This needs to be done ASAP.

How would you differentiate, that you are upgrading from Fedora 16?

Anyway, I think we should remove this script altogether in both Fedora 19 and Fedora 20. Even Fedora 17 is already end of life, so I do not think we are expected to support upgrades from pre-EOL Fedora to 19 or 20.
Comment 4 Nathan Kinder 2013-08-14 12:43:16 EDT
(In reply to Martin Kosek from comment #1)
> I noticed the "/var/lock" symlinks in dirsrv's tmpfiles config file:
> 
>  # cat /etc/tmpfiles.d/dirsrv-IDM-LAB-BOS-REDHAT-COM.conf 
> d /var/run/dirsrv 0770 dirsrv dirsrv
> d /var/lock/dirsrv 0770 dirsrv dirsrv
> d /var/lock/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM 0770 dirsrv dirsrv
> 
> However, is this something that IPA should tackle? It is created during
> setup-ds.pl phase, by DS. I would expect that DS would update it during
> upgrade (if necessary). Rich, please advise if this is something that IPA is
> expected to do.

This does seem to be some sort of race condition, as a fresh install of 389-ds-base on F19 works fine on a reboot in my testing.  We could change this to use /run/lock instead of /var/lock in our tmpfiles.d config in DS, but I don't understand why that should be necessary.  I would like to understand what is happening before making this change.

The fact that using /run/lock instead of /var/lock is working around the  problem indicates that /usr/lib/tmpfiles.d/legacy.conf is being processed before /etc/tmpfiles.d/dirsrv.conf, as the /run/lock directory must already exist.  This doesn't explain why /var/lock wouldn't work, as /var/lock is a persistent symlink on a normal filesystem as far as I can tell.  If /run/lock exists, /var/lock should work just fine.
Comment 5 Bill Peck 2013-08-14 12:57:14 EDT
If someone can tell me how to gather more data or enable certain debugging from systemd to try and shed some light on the issue I'd be more than happy to do it.

Should the systemd developers be cc'ed on this?

Thanks
Comment 6 Martin Kosek 2013-08-15 02:24:57 EDT
Good idea. Adding Michal of systemd to chime in if this area is familiar to him.
Comment 7 Michal Sekletar 2013-08-21 12:24:45 EDT
First of all sorry for not commenting earlier. I think we are hitting an actual bug in systemd-tmpfiles here, since using /run/lock works and /var/lock doesn't. 

Just fyi, tmpfiles drop-in configuration is applied in a way that config files in /etc override those in /usr and after parsing all config files, configuration is then applied according to lexicographical ordering of config file names, but regardless of location.

This means that /etc/tmpfiles.d/dirsrv* config files are applied earlier than /usr/lib/tmpfiles.d/legacy.conf. However, this *shouldn't* matter since we always create whole hierarchy for each path specified in config files and if there is overlap with configuration in later config files then path modes are fixed-up accordingly.

AFAIK, the bug is that that we don't canonicalize path before we start with creating parent hierarchy for path specified in configuration file.

Please use /run paths as a workaround for the time being. Although you might want to keep those and stop using legacy /var paths.

Btw, freeipa-server ships /etc/tmpfiles.d/ipa.conf in package payload. Note that you shouldn't do that as per Fedora Packaging Guidelines policy.
Comment 8 Rob Crittenden 2013-08-27 09:28:58 EDT
(In reply to Michal Sekletar from comment #7)
> Btw, freeipa-server ships /etc/tmpfiles.d/ipa.conf in package payload. Note
> that you shouldn't do that as per Fedora Packaging Guidelines policy.

I'm not sure I follow. It looks like we are incorrectly marking it as config, and using the wrong directory, is that what you mean?
Comment 9 Michal Sekletar 2013-08-27 11:31:39 EDT
Sorry for confusing statement. I was referring to the path /etc/tmpfiles.d/ipa.conf.

As you correctly pointed out, tmpfiles drop-in shouldn't be in /etc/ and shouldn't be marked as config.
Comment 10 Martin Kosek 2013-09-02 11:25:11 EDT
Thanks, I will create an upstream ticket to fix the ipa.conf drop-in and place it in the right directory to avoid violating the guidelines.
Comment 11 Martin Kosek 2013-09-02 11:31:12 EDT
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3894
Comment 12 Martin Kosek 2013-09-03 10:30:01 EDT
Upstream ticket in Comment 11 was closed as a duplicate of:
https://fedorahosted.org/freeipa/ticket/3881
Comment 13 Orion Poplawski 2013-09-13 09:36:35 EDT
I can edit /etc/tmpfiles.d/dirsrv@NWRA-COM, but then something seems to be changing it back (perhaps updates of 386-ds/ipa).
Comment 14 Martin Kosek 2013-09-16 02:20:20 EDT
(In reply to Orion Poplawski from comment #13)
> I can edit /etc/tmpfiles.d/dirsrv@NWRA-COM, but then something seems to be
> changing it back (perhaps updates of 386-ds/ipa).

I think this is owned by 389-ds-base. FreeIPA does not touch it. I filed a Bug 1008306 to get it fixed.
Comment 16 Adam Williamson 2013-09-27 12:18:55 EDT
*** Bug 983073 has been marked as a duplicate of this bug. ***
Comment 17 Adam Williamson 2013-09-27 12:27:16 EDT
Orion: Martin: instead of modifying the existing file, you could just create a new one with the relevant directives, I guess. I think it wouldn't matter whether the old one happened to succeed or not on a particular boot.

Picking up some earlier threads:

"How would you differentiate, that you are upgrading from Fedora 16?"

The standard way to do this is to check the version of the package from which you're upgrading, assuming this is done in RPM scriptlets. It's a pretty normal thing to do. Or you can have some other kind of obvious 'sanity check', like 'does all the stuff I would do if I was run look like it was already done?'

"Even Fedora 17 is already end of life, so I do not think we are expected to support upgrades from pre-EOL Fedora to 19 or 20."

You're not _expected_ to, but it's kind of nice if you can. People _do_ sometimes run systems past EOL and then try to upgrade them, and if anything, it tends to be more likely to be big complex configs they were scared to upgrade.
Comment 18 Martin Kosek 2013-09-30 02:20:04 EDT
(In reply to Adam Williamson from comment #17)
> Orion: Martin: instead of modifying the existing file, you could just create
> a new one with the relevant directives, I guess. I think it wouldn't matter
> whether the old one happened to succeed or not on a particular boot.

Right. But since we plan to release soon, I think it makes sense to just fix the config file and not have it twice. At least for config file sanity.
Comment 19 Fedora Update System 2013-10-04 10:51:52 EDT
freeipa-3.3.2-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/freeipa-3.3.2-1.fc19
Comment 20 Fedora Update System 2013-10-05 21:32:11 EDT
Package freeipa-3.3.2-1.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing freeipa-3.3.2-1.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-18371/freeipa-3.3.2-1.fc19
then log in and leave karma (feedback).
Comment 21 Fedora Update System 2013-10-14 02:58:23 EDT
freeipa-3.3.2-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 22 Fedora Update System 2013-10-14 13:09:21 EDT
freeipa-3.3.2-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.