For openQA purposes, we are using virt-builder to build Fedora disk images with various package sets. One of them has the Workstation package group installed. We achieve this by running virt-builder with this commands file: root-password password:weakpassword update selinux-relabel install @workstation-product-environment link /usr/lib/systemd/system/graphical.target:/etc/systemd/system/default.target firstboot-command useradd -m -p '' ejohn firstboot-command echo 'ejohn:weakpassword' | chpasswd which should be pretty self-explanatory. The 'selinux-relabel' tells virt-builder to try and relabel the system after it does all the other stuff (because the way virt-builder works, it messes up all the SELinux labelling), but virt-builder can't always do this live in its little appliance VM. When it can't, it instead creates /.autorelabel , so first boot of a VM with the disk image attached should cause the relabel to occur. When I create a Fedora 23 image using this commands file: LIBGUESTFS_BACKEND=direct virt-builder fedora-23 -o f23.img --arch x86_64 --size 21474836480b --commands-from-file /root/openqa_fedora_tools/tools/desktop.commands that works fine. If I boot the image in qemu - `qemu-kvm -m 2G -nographic f23.img` - there are various obviously-label-related issues, but it does manage to reach fedora-autorelabel.service and do the relabel. Then it reboots, successfully. However, if I create a Fedora 24 image in exactly the same way: LIBGUESTFS_BACKEND=direct virt-builder fedora-24 -o f24.img --arch x86_64 --size 21474836480b --commands-from-file /root/openqa_fedora_tools/tools/desktop.commands then try to boot it with `qemu-kvm -m 2G -nographic f24.img`, it never reaches the relabel service, apparently. Instead it seems to dump me into the rescue prompt. Further, as mentioned, we build several images with different package sets. Another one we build is 'minimal', using this commands file: root-password password:weakpassword update selinux-relabel so we just keep the package set from the base image, but tell it to run a 'dnf update' and trigger a relabel. *That* one does work for F24. If I build that image: LIBGUESTFS_BACKEND=direct virt-builder fedora-24 -o f24-minimal.img --arch x86_64 --size 21474836480b --commands-from-file /root/openqa_fedora_tools/tools/minimal.commands and boot it with `qemu-kvm -m 2G -nographic f24-minimal.img`, it *does* reach the relabel service and relabel the disk properly. So it seems like, somehow, the problem kicks in on installation of some package (presumably containing a systemd service) or other, that's in the deps of the workstation package set. Or I *guess* it could be the way default.target is changed to graphical.target on the workstation image, but I don't see why that would matter. I will attach the console output from booting all three situations (f23 desktop image, f24 desktop image, f24 minimal image) for comparison. I'll also see if I can get systemd debug output...
Created attachment 1171631 [details] console output when booting f23-desktop image (relabel works)
Created attachment 1171632 [details] console output when booting f24-desktop image (relabel fails)
Created attachment 1171633 [details] console output when booting f24-minimal image (relabel works)
An obvious difference is the 'Update is Completed' service (which is systemd-update-done.service ). On f23-desktop and f24-minimal this completes successfully just before the relabel. On f24-desktop it fails, and this is the point at which we get dropped to the rescue prompt. The service is 'wanted' by sysinit.target. Still, several other services 'wanted' by sysinit.target fail in all cases (e.g. systemd-journald.service and systemd-sysctl.service , just for example) so I don't know what's special about systemd-update-done.service which would make it fatal. I guess we'll need those debug logs.
Created attachment 1171643 [details] console output when booting f24-desktop image (relabel fails) with systemd debugging enabled OK, here's the systemd debug output from booting the problematic case. One particular chunk leaps out at me: Activating default unit: default.target default.target: Failed to load configuration: Permission denied Failed to load default target: Permission denied Trying to load rescue target... which suggests it gives up on default.target and picks rescue.target instead, and I guess that's why the relabel never runs; relabel is wanted by basic.target , not sysinit.target , and rescue.target wants sysinit.target but not basic.target . So we never hit basic.target , thus we never hit relabel. So I guess the change of default.target in desktop.commands actually is what triggers this? There is an AVC for default.target earlier in the log: [ 107.137874] audit: type=1400 audit(1466707575.241:62): avc: denied { read } for pid=1 comm="systemd" name="default.target" dev="sda3" ino=49798 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file permissive=0 so all that kinda adds up, but then I dunno why it works in F23. I guess I'll have to get the debug log for the F23 case and see how it compares.
Created attachment 1171698 [details] console output when booting f23-desktop image (relabel works) with systemd debugging enabled Here's the F23 debug log. It looks like there's no denial for default.target , so it gets loaded fine. So I guess systemd is more tightly confined in F24, or something? Not really sure how to 'fix' this, since we seem to be in a nexus between virt-builder and systemd and I guess selinux-policy , and I'm not sure who considers what to be 'supported'...
adding lvrabec to the party.
so I found a fairly simple way to work around this: make the default.target change a firstboot-command (so it gets run on the first 'normal' boot of the generated image, when SELinux is loaded normally and so the label winds up correct) instead of having it done in the virt-builder appliance at image build time (so the label winds up wrong). Now I'm fighting another weird systemd issue, sigh... I'll leave this open for now just in case anyone thinks it's a bug they want to fix, but if no-one does, we can call it an unfortunate interaction and close it.
This message is a reminder that Fedora 24 is nearing its end of life. Approximately 2 (two) weeks from now Fedora will stop maintaining and issuing updates for Fedora 24. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '24'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 24 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.