Description of problem: Fedora-Silverblue-ostree-x86_64-Rawhide-20200715.n.2.iso The following error occurred while installing. This is a fatal error and installation will be aborted. systemd-tmpfiles ['--create', '--boot', '--root=/mnt/sysroot', '--prefix=/var/spool'] exited with code 65 Version-Release number of selected component (if applicable): anaconda 33.22-1.fc33 How reproducible: Always Steps to Reproduce: 1. Do a default installation 2. 3. Actual results: Fails Expected results: Success Additional info: # chroot /mnt/sysimage chroot: failed to run command '/bin/sh': No such file or directory
Created attachment 1701327 [details] anaconda.log
Created attachment 1701328 [details] journal
Created attachment 1701329 [details] program.log
Created attachment 1701330 [details] storage.log
Might be related to new systemd v246~rc1-1.fc33
Created attachment 1701334 [details] journalctl with log_level debug trimmed the top off; log starts at rpmostree install start, doesn't seem super revealing what's gone wrong though
[ 532.201613] fedora anaconda[1544]: program: Running... systemd-tmpfiles --create --boot --root=/mnt/sysroot --prefix=/var/spool [ 532.270176] fedora anaconda[1544]: program: /mnt/sysroot/usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf:132: Failed to resolve group 'mail'. [ 532.271034] fedora anaconda[1544]: program: /mnt/sysroot/usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf:134: Failed to resolve group 'lp'. [ 532.271653] fedora anaconda[1544]: program: /mnt/sysroot/usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf:135: Failed to resolve group 'lp'. [ 532.273101] fedora anaconda[1544]: program: /mnt/sysroot/usr/lib/tmpfiles.d/var.conf:23: Duplicate line for path "/mnt/sysroot/var/spool", ignoring. [ 532.273923] fedora anaconda[1544]: program: Return code: 65 65 means that the config was invalid. There is a change in systemd-246 that passwd database from the root is used instead of the one from host. Does /mnt/sysroot/etc/group contain mail and lp?
No. Off hand since this is an rpm-ostree installation, I think it's expected that /etc is mostly empty? # cat /mnt/sysroot/etc/group root:x:0: wheel:x:10: # ls /mnt/sysimage/etc/ resolv.conf # df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 1.4G 0 1.4G 0% /dev tmpfs 290M 12K 290M 1% /dev/shm tmpfs 579M 23M 556M 4% /run /dev/sr0 2.7G 2.7G 0 100% /run/install/repo /dev/mapper/live-rw 7.9G 6.7G 1.2G 86% / tmpfs 290M 988K 289M 1% /tmp /dev/vda3 99G 2.4G 95G 3% /mnt/sysimage /dev/vda2 976M 91M 819M 10% /mnt/sysimage/boot /dev/vda1 599M 8.5M 591M 2% /mnt/sysimage/boot/efi tmpfs 1.5G 0 1.5G 0% /mnt/sysroot/dev/shm /dev/vda3 99G 2.4G 95G 3% /mnt/sysimage/home
Also happens with Fedora-IoT-IoT-ostree-x86_64-33-20200716.0.iso
Likely related to rpm-ostree using nss-altfiles, so all the system users/groups are actually in /usr/lib/{passwd,group}. Any user/group mentioned in tmpfiles not in the Anaconda environment (or with the new systemd behaviour, if nss-altfiles isn't used in the chroot) will cause lookup errors. See the relevant Anaconda comment block about this: https://github.com/rhinstaller/anaconda/blob/71ac3af3f46aae7cd17fe48ac41537f0c31f7e56/pyanaconda/payload/rpmostreepayload.py#L352-L359. That said, I don't quite remember now how this worked even before v246; presumably e.g. `/var/spool/cups` was still owned by `lp` so that should've caused a lookup failure too. Unless it was in Anaconda's /etc/group? Long-term we're looking to switch over to systemd-sysusers: https://github.com/coreos/rpm-ostree/issues/49 https://github.com/coreos/rpm-ostree/pull/1679 So then the Anaconda code there would have to first run `systemd-sysusers` before running `systemd-tmpfiles`.
> I don't quite remember now how this worked even before v246 systemd-tmpfiles would resolved uids/gids in the host system. That doesn't sound too good, but things like 'mail' and 'lp' quite often have the same numbers in different systems so in practice it wasn't that bad. It sounds like the the code used by systemd-tmpfiles should be tweaked to look at /usr/lib/{passwd,group} if /etc/{passwd,group} are not found. This would be very easy to implement. I can prep a patch if you think this would work for you.
Please see https://github.com/systemd/systemd/pull/16512.
I am changing the component to systemd based on the comment 12.