Bug 1857530

Created attachment 1701327 [details]
anaconda.log

Created attachment 1701328 [details]
journal

Created attachment 1701329 [details]
program.log

Created attachment 1701330 [details]
storage.log

Might be related to new systemd v246~rc1-1.fc33

Created attachment 1701334 [details]
journalctl with log_level debug

trimmed the top off; log starts at rpmostree install start, doesn't seem super revealing what's gone wrong though

[  532.201613] fedora anaconda[1544]: program: Running... systemd-tmpfiles --create --boot --root=/mnt/sysroot --prefix=/var/spool
[  532.270176] fedora anaconda[1544]: program: /mnt/sysroot/usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf:132: Failed to resolve group 'mail'.
[  532.271034] fedora anaconda[1544]: program: /mnt/sysroot/usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf:134: Failed to resolve group 'lp'.
[  532.271653] fedora anaconda[1544]: program: /mnt/sysroot/usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf:135: Failed to resolve group 'lp'.
[  532.273101] fedora anaconda[1544]: program: /mnt/sysroot/usr/lib/tmpfiles.d/var.conf:23: Duplicate line for path "/mnt/sysroot/var/spool", ignoring.
[  532.273923] fedora anaconda[1544]: program: Return code: 65

65 means that the config was invalid. There is a change in systemd-246 that passwd database from the
root is used instead of the one from host. Does /mnt/sysroot/etc/group contain mail and lp?

No. Off hand since this is an rpm-ostree installation, I think it's expected that /etc is mostly empty?

# cat /mnt/sysroot/etc/group
root:x:0:
wheel:x:10:

# ls /mnt/sysimage/etc/
resolv.conf

# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             1.4G     0  1.4G   0% /dev
tmpfs                290M   12K  290M   1% /dev/shm
tmpfs                579M   23M  556M   4% /run
/dev/sr0             2.7G  2.7G     0 100% /run/install/repo
/dev/mapper/live-rw  7.9G  6.7G  1.2G  86% /
tmpfs                290M  988K  289M   1% /tmp
/dev/vda3             99G  2.4G   95G   3% /mnt/sysimage
/dev/vda2            976M   91M  819M  10% /mnt/sysimage/boot
/dev/vda1            599M  8.5M  591M   2% /mnt/sysimage/boot/efi
tmpfs                1.5G     0  1.5G   0% /mnt/sysroot/dev/shm
/dev/vda3             99G  2.4G   95G   3% /mnt/sysimage/home

Also happens with Fedora-IoT-IoT-ostree-x86_64-33-20200716.0.iso

Likely related to rpm-ostree using nss-altfiles, so all the system users/groups are actually in /usr/lib/{passwd,group}. Any user/group mentioned in tmpfiles not in the Anaconda environment (or with the new systemd behaviour, if nss-altfiles isn't used in the chroot) will cause lookup errors. See the relevant Anaconda comment block about this: https://github.com/rhinstaller/anaconda/blob/71ac3af3f46aae7cd17fe48ac41537f0c31f7e56/pyanaconda/payload/rpmostreepayload.py#L352-L359.

That said, I don't quite remember now how this worked even before v246; presumably e.g. `/var/spool/cups` was still owned by `lp` so that should've caused a lookup failure too. Unless it was in Anaconda's /etc/group?

Long-term we're looking to switch over to systemd-sysusers:
https://github.com/coreos/rpm-ostree/issues/49
https://github.com/coreos/rpm-ostree/pull/1679

So then the Anaconda code there would have to first run `systemd-sysusers` before running `systemd-tmpfiles`.

> I don't quite remember now how this worked even before v246

systemd-tmpfiles would resolved uids/gids in the host system. That doesn't sound too good, but
things like 'mail' and 'lp' quite often have the same numbers in different systems so in
practice it wasn't that bad.

It sounds like the the code used by systemd-tmpfiles should be tweaked to look
at /usr/lib/{passwd,group} if /etc/{passwd,group} are not found. This would be very
easy to implement. I can prep a patch if you think this would work for you.

Please see https://github.com/systemd/systemd/pull/16512.

I am changing the component to systemd based on the comment 12.