Chrony requires "/var/lib/chrony" for its working space. In the bootc image creation context, /var/lib/* is often cleaned up[1] After installation with bootc-imaged Fedora, chronyd systemd service fails to start because of lack of /var/lib/chrony with very simple error. --- Jun 16 02:53:30 bootc-test-001 (chronyd)[3476]: chronyd.service: Failed to set up mount namespacing: /var/lib/chrony: No such file or directory Jun 16 02:53:30 bootc-test-001 (chronyd)[3476]: chronyd.service: Failed at step NAMESPACE spawning /usr/sbin/chronyd: No such file or directory --- Of course I understand it's not a bug of chrony and package. We should take any way to inject this directory in/after installation as workaround. Currently I'll use custom conf for systemd-tmpfiles. I've confirmed that just add a simple conf file to /etc/tmpfiles.d/chrony.conf as following. --- "/etc/tmpfiles.d/chrony.conf" d /var/lib/chrony 0750 chrony chrony - - --- If this idea is acceptable, can we add this file to the Fedora(and CentOS Stream/RHEL also) packaging? I'll be happy if I can see any thoughts about this. [1] https://docs.fedoraproject.org/en-US/bootc/building-from-scratch/ Reproducible: Always Steps to Reproduce: 1. Create fedora bootc image from scratch with this procesure. In the Containerfile, "RUN dnf install -y chrony" https://docs.fedoraproject.org/en-US/bootc/building-from-scratch/ 2. Boot with the image which is created above(for example, install to baremetal system with anaconda-iso by bootc-image-builder) https://docs.fedoraproject.org/en-US/bootc/bare-metal/ 3. Check if chronyd.service is running or not. Actual Results: chronyd.service failed to start with the error "chronyd.service: Failed to set up mount namespacing: /var/lib/chrony: No such file or directory" Expected Results: chronyd.service start successfully.
Is chrony the only impacted package? To me this looks like something that should be handled in a more general way than patching individual packages to be able to recover from a missing directory.
Hi Miroslav, Thank you for your comment. No. Not only chrony but other packages will be impacted in this context. Just in my use case I've found that chrony and some packages are hit to this problem on earlier timing. Totally I agree with you. The behavior and policy for missing file/directory depends on each software developer/project. I think it's good, no problem. This is just my opinion. I think that systemd-tmpfiles mechanism provides a certain level unified way to handle missing files/directories for each software. Of course I don't think it should be mandated to every software. So I'd like to have comments from maintainers/developers. If it doesn't feel good to you. Please close it with WONTFIX. Thank you for taking your time to consider it.
There are indeed more general ideas on trying to address this at build time (see e.g. https://github.com/bootc-dev/bootc/issues/1055), but the approaches there are more of a workaround. The preference is indeed that packages ship tmpfiles.d dropins to populate `/var`. See also https://pagure.io/packaging-committee/pull-request/1333.
> To me this looks like something that should be handled in a more general way than patching individual packages to be able to recover from a missing directory. The best fix actually is to drop the directory from the RPM (keep it as %ghost if you want) and then it's not "recover[ing] from a missing directory" it's that the directory is *always* created via tmpfiles.d consistently. These directories are also owned by non-root users, xref https://github.com/bootc-dev/bootc/issues/1263 for uid-gid drift issues. If these were dropped, then everything in the RPM would be owned by root, and hence everything in the container image would be, which dramatically simplifies uid/gid management.
If the directory wasn't included in the rpm, it wouldn't be created on the package installation, only after reboot, when tmpfiles is executed, right? Or is there some rpm magic that triggers tmpfiles? How about specifying the directory as StateDirectory in the chronyd unit itself? That's already used in the chronyd-restricted service, so that would be more consistent. This should also work with %ghost.
This should be now addressed in chrony-4.7-2.fc43. The chronyd unit file specifies the log and lib directories, so they don't need to exist and in the rpm they are marked as ghosts.
> How about specifying the directory as StateDirectory in the chronyd unit itself? Yes I forgot to mention that one - it's indeed even better to do that especially because it works nicely with DynamicUser=yes which I think chrony could use instead of a static user right? Which hmm, relating to your patch in https://src.fedoraproject.org/rpms/chrony/c/53321f84b8e15e39e3483fcc5f396bc4e4244e42?branch=rawhide crony and specifically use of `!` in ExecStart...is chrony changing uid itself? AFAIK the use of User= will be ignored for the main binary. > If the directory wasn't included in the rpm, it wouldn't be created on the package installation, only after reboot, when tmpfiles is executed, right? Or is there some rpm magic that triggers tmpfiles? Yes, I think there's some usage of %post for this.
(In reply to Colin Walters from comment #7) > especially because it works nicely with DynamicUser=yes which I think chrony > could use instead of a static user right? If it didn't need to write and read its own files (e.g. NTS cookies/keys), I think yes. > > Which hmm, relating to your patch in > https://src.fedoraproject.org/rpms/chrony/c/ > 53321f84b8e15e39e3483fcc5f396bc4e4244e42?branch=rawhide crony and > specifically use of `!` in ExecStart...is chrony changing uid itself? AFAIK > the use of User= will be ignored for the main binary. Depending on the configuration, chronyd may need root privileges to open and configure various devices (e.g. RTC, PPS, PHC). It drops the root privileges after start before any network activity. If the configuration doesn't require the privileges, chronyd can be started under the non-root user directly. That's what the chronyd-restricted service is for.
> If it didn't need to write and read its own files (e.g. NTS cookies/keys), Those could be in StateDirectory no? > Depending on the configuration, chronyd may need root privileges to open and configure various devices (e.g. RTC, PPS, PHC). There's the `Open` directive that may help avoid that.
How would it read files that were created by a different user? The UID/GID would not be stable across restarts, right? I think it would be annoying if the devices needed in the specific configuration had to be specified also in the service file and passed to chronyd.
> How would it read files that were created by a different user? The UID/GID would not be stable across restarts, right? With DynamicUser=yes systemd will chown StateDirectory if necessary. > I think it would be annoying if the devices needed in the specific configuration had to be specified also in the service file and passed to chronyd. Yes; in theory one could get around this with a generator that parsed the config file and added a drop-in for the unit. That said note that OpenFile= supports "graceful" which will handle ENOENT, so a simple option is to just try to open all of them by default.
(In reply to Colin Walters from comment #11) > With DynamicUser=yes systemd will chown StateDirectory if necessary. Interesting. I wasn't aware of that. So basically the admin would have to keep a copy or directly maintain the keys (what is normally in /etc/chrony.keys and /etc/pki/tls) in the StateDirectory, or is there some systemd mechanism that could do that automatically? > > I think it would be annoying if the devices needed in the specific configuration had to be specified also in the service file and passed to chronyd. > > Yes; in theory one could get around this with a generator that parsed the > config file and added a drop-in for the unit. Any examples of services in Fedora doing that?
> (what is normally in /etc/chrony.keys and /etc/pki/tls) Can't we use OpenFile= for the key file at least? Then it can just be owned by root, and that concern goes away. For /etc/pki/tls, most of the time in my experience (since these are public keys) admins just make them world readable anyways right?
OpenFile= won't work as chronyd needs to be able to reload the files when the "chronyc rekey" command is issued. To configure an NTS server a private key is required. That cannot be world readable.
> OpenFile= won't work as chronyd needs to be able to reload the files when the "chronyc rekey" command is issued. Ah yes, that indeed makes things a lot more complex. Would it be too problematic to just have `rekey` restart the service? Otherwise we'd need something like a privileged service run in the same unit that opens the file and then can update the systemd fdstore and the unprivileged process reads it.
> To configure an NTS server a private key is required. That cannot be world readable. If this is a directory then another general pattern here is to have e.g. /etc/pki/private or so with mode 0700 (an "inaccessible directory") and then use BindReadOnlyPaths=/etc/pki/private/chrony:/run/chrony/pki or so.