Fedora 38 and Rawhide live images since yesterday's compose show a warning message on boot: "Unable to find a persistent overlay; using a temporary one. All root filesystem changes will be lost on shutdown. Press [Enter] to continue." This is a result of recent changes to overlay stuff, Neal asked me to file the bug here, so here it is. Arguably this ought to be a release blocker, but for an oversight in the criteria. We have this wording regarding *installed system* boot: "In all of the above cases, the boot should proceed without any unexpected user intervention being required." but we don't, for some reason, have the same stipulation for *deployment media* boot, which seems weird. If we're against unexpected interaction in the one case, we should be against it in the other. I'll maybe bring this up on test@.
Well, seems it's worse than that. In openQA, even after hitting enter, boot is failing most of the time, or occasionally reaching a login prompt instead of a desktop. On a local VM boot does seem to work, but `livesys.service` fails to start, and running a console gives a wrong terminal prompt (bash-5.2$), which indicates the user account wasn't set up properly. ausearch shows a flood of AVCs like this: ---- time->Thu Feb 16 17:35:55 2023 type=AVC msg=audit(1676586955.315:143): avc: denied { create } for pid=1065 comm="useradd" name="liveuser" scontext=system_u:system_r:kernel_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir permissive=0 ---- time->Thu Feb 16 17:35:55 2023 type=AVC msg=audit(1676586955.652:150): avc: denied { create } for pid=1 comm="systemd" name="#42" scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=chr_file permissive=0 ---- time->Thu Feb 16 17:35:55 2023 type=AVC msg=audit(1676586955.688:151): avc: denied { create } for pid=1 comm="systemd" name="#43" scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=chr_file permissive=0 ---- time->Thu Feb 16 17:35:55 2023 type=AVC msg=audit(1676586955.940:157): avc: denied { create } for pid=1171 comm="rm" name="#4a" scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=chr_file permissive=0 ---- (there are lots more, those are just the first few). If I edit the boot params and drop all the overlay-related ones, all services start successfully and the console prompt is normal.
These effects happen whether the image is attached as an emulated optical drive or emulated USB stick. Haven't tested on metal with a real USB stick yet.
Same behaviour on a real system with a real USB stick (prompt on boot, failed services, bad console prompt).
The change has been reverted upstream and downstream for now, but leaving the bug report open as that's obviously not the long-term fix.
Created attachment 1944865 [details] journalctl log from first boot
Problems I've noticed so far: These prevent the autooverlay boot: 1. dracut-live is missing ../90overlayfs module -- Where is that composed? 2. src/pylorax/creator.py is missing dmsquash-live-autooverlay in DRACUT_DEFAULT With those corrections made via a patched livecd-creator --base-on --shell edit session, I was able to boot a Fedora SoaS 38 LiveUSB dd'ed from the .iso with autooverlay by setting enforcing=0 in the kernel command line. Evenso, 1. dmraid-activation.service failed for want of missing /etc/init.d/functions 2. polkit.service fails to start due to access denials 3. avahi-daemon.service fails to start due to access denials See the attached journalctl log.
This also should be part of the first list of items preventing autooverlay boot: 3. Irregular GPT partition table, see these lines: [ 4.530459] fedora kernel: scsi 8:0:0:0: Direct-Access Generic Flash Disk 8.07 PQ: 0 ANSI: 4 [ 4.530903] fedora kernel: sd 8:0:0:0: Attached scsi generic sg3 type 0 [ 4.531804] fedora kernel: sd 8:0:0:0: [sdc] 3934208 512-byte logical blocks: (2.01 GB/1.88 GiB) [ 4.532557] fedora kernel: sd 8:0:0:0: [sdc] Write Protect is off [ 4.532582] fedora kernel: sd 8:0:0:0: [sdc] Mode Sense: 23 00 00 00 [ 4.533258] fedora kernel: sd 8:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [ 4.551675] fedora kernel: GPT:Primary header thinks Alt. header is not at the end of the disk. [ 4.551690] fedora kernel: GPT:2468151 != 3934207 [ 4.551697] fedora kernel: GPT:Alternate GPT header not at the end of the disk. [ 4.551705] fedora kernel: GPT:2468151 != 3934207 [ 4.551712] fedora kernel: GPT: Use GNU Parted to correct GPT errors. I had to add --fix to the parted --script parameters in 90dmsquash-live-autooverlay/create-overlay.sh at line 74: 74 freeSpaceStart=$(parted --script --fix ${blockDevice} unit % print free \
Created attachment 1945372 [details] live.debug log for create-overlay
Created attachment 1945373 [details] live.debug log for dmsquash-live-root
Created attachment 1945374 [details] journalctl log for auto-overlay boot
https://github.com/dracutdevs/dracut/pull/2215 includes changes to allow autooverlay booting with * enforcing=0 * an edited image to include the 90overlayfs dracut module in dracut-live, Where is dracut-live composed? * and adjusting the dracut configuration arguments to include dmsquash-live-autooverlay (https://github.com/weldr/lorax/pull/1308/files#r1112398922 shows the change needed.) The attached journalctl log for Fedora-Workstation-Live-x86_64-38-20230215.n.0.iso (updated with dnf upgrade) shows that SELinux configuration is still lacking. Who can debug the SELinux denials?
Dropping this from the FE list as it's been reverted for now.
This message is a reminder that Fedora Linux 38 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 38 on 2024-05-21. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '38'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 38 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Neal says this was left out of F39 and F40 but can be re-tested with current Rawhide, so bumping it to Rawhide. I'll re-test.
I'm not sure this is really testable yet, I don't think https://fedoraproject.org/wiki/Changes/ModernizeLiveMedia has really been implemented. persistence does not seem to be enabled by default and there are no boot menu entries dealing with it (as the Change page says there should be). At least I can say current Rawhide Workstation live boots fine with no *errors* caused by persistence stuff, but that's all.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 42 development cycle. Changing version to 42.