Bug 2246410

Here are videos of my experience on bare metal:

Desktop PC: https://youtu.be/YBNO_4DwHUk
Laptop: https://youtu.be/NnKwbR5szeA

Created attachment 1995678 [details]
media check failure in a VM

This is a screenshot of how the display should look like - and how it looks in a VM (no black screen, stays visible forever).

Proposing for a blocker discussion:
"Validation of install media must work correctly for all release-blocking images. 
This means that the installer's mechanism for verifying that the install medium is intact must complete successfully if the medium is correctly written and return a legible failure message if it is not."
https://fedoraproject.org/wiki/Fedora_39_Final_Release_Criteria#Media_consistency_verification

This problem seems to be a new regression in F39. With a F38 Workstation Live image (when corrupted as described in the reproducer), the error message stays printed on the screen, the screen doesn't turn off.

This happens in both UEFI and BIOS boot modes, at least for me.

I can reproduce on two laptops, happens the same on both every time. Failure is just like the 2nd video (laptop) in c1. If I'm paying attention to the boot, I can see the red text FAIL. And even when I see the FAIL, I'm inferring that it's related to the media checker, it's too fast to know that for sure if I'm not expecting it in advance.

yeah, confirmed here too on my test desktop.

I'm gonna take a guess at this being systemd?

Or, maybe dracut. That's where the check is implemented. This way everybody is CCed...

It  might be useful to get a full log somehow, and see if we can figure out what is making the screen go black. Maybe it's not the kernel. It could be plymouth still loads, and switching to graphics mode, but then the rest of the boot gets stopped by the failed media check? It's pretty easy to get logs out of a VM, but harder with Lives to get logs out once the screen goes black. If the problem reproduces with DVD or netinstall, enabling the ssh feature of anaconda without password would make it straightforward to extract with 'journalctl -b -o short-monotonic --no-pager' etc.

Makes me wonder if 'rd.break=pre-mount' is still working, i.e. is it possible non-Live install boot failures get stuck in this same situation and we can't even use a dracut shell to troubleshoot? (or any rd.break option for that matter)

sorry: non-Live & non-install, ergo booting a (post-install) system.

dracut specifically turns off the plymouth splash for the duration of the media check. this is the dracut code:

if [ -n "$check" ]; then
    type plymouth > /dev/null 2>&1 && plymouth --hide-splash
    if [ -n "$DRACUT_SYSTEMD" ]; then
        p=$(dev_unit_name "$check_dev")
        systemctl start checkisomd5@"${p}".service
    else
        checkisomd5 --verbose "$check_dev"
    fi
    if [ $? -eq 1 ]; then
        die "CD check failed!"
        exit 1
    fi
    type plymouth > /dev/null 2>&1 && plymouth --show-splash
fi

it has not changed since 2014.

the checkisomd5 service is also a part of dracut and, similarly, has not changed since 2014.

When I add "nomodeset", the bug is not triggered, the display stays active.

Discussed at 2023-10-26 go/no-go meeting acting as a blocker review meeting - https://meetbot-raw.fedoraproject.org/fedora-meeting/2023-10-26/ . Accepted as a blocker as a clear violation of "Validation of install media must work correctly for all release-blocking images. ... This means that the installer's mechanism for verifying that the install medium is intact must complete successfully if the medium is correctly written and return a legible failure message if it is not."

(In reply to Kamil Páral from comment #14)
> When I add "nomodeset", the bug is not triggered, the display stays active.

OK, probably kernel? I'm not offhand thinking of anything else that reacts to that.

it's definitely the kernel that responds to nomodeset , but the fact that disabling modesetting avoids the problem doesn't necessarily mean it's the kernel *causing* it. (disabling modesetting can also work around bugs in mutter, for instance, but the bug is still in mutter).

still, we can cc jforbes. he was at the blocker meeting so he's probably aware already, but hey.

This is also reproducible with an Everything Netinst image. Furthermore, I've verified that the bug is not present with F38 main repo netinst image, but it is present with F38 main+updates repos netinst image. So the regression is caused by something in F38 updates. I'll try to nail down what it is.

That was simple - just building an image with the latest F38 kernel (and all other packages from the main repo) exhibits the bug.

So we have:
kernel-6.2.9-300.fc38 - OK
kernel-6.5.8-200.fc38 - BROKEN

I guess I'll run through all kernels available in Koji to narrow down the scope.

Given that nomodeset works around it, is this only impacting specific hardware? Is it amd, nvidia, i915?

It appears that my information in comment 18 and 19 might have been incorrect. Sorry, I'll post when I know more.

(In reply to Justin M. Forbes from comment #20)
> Given that nomodeset works around it, is this only impacting specific
> hardware? Is it amd, nvidia, i915?

In my case, the desktop PC has Radeon 580 and the laptop has "Intel Corporation CometLake-H GT2 [UHD Graphics] (rev 05)".

(In reply to Kamil Páral from comment #21)
> It appears that my information in comment 18 and 19 might have been
> incorrect. Sorry, I'll post when I know more.

Yeah, I was wrong. Here are my current notes:

= Live images =

Fedora-Workstation-Live-x86_64-38-1.6.iso  -- OK
F38-WORK-x86_64-LIVE-20231016.iso (live respin [1]) -- OK

Fedora-Workstation-Live-x86_64-39-1.2.iso -- BROKEN
Fedora-Workstation-Live-x86_64-39_Beta-1.1.iso -- BROKEN

Anything that I build on my F39 system using livecd-creator (including: F38 main repo, F38 main+updates repo) -- BROKEN

= Netinst =

Fedora-Everything-netinst-x86_64-38-1.6.iso  -- BROKEN
Fedora-Everything-netinst-x86_64-39_RC-1.2.iso  -- BROKEN

Anything that I build on my F39 system using lorax -- BROKEN

~~~~

So, this issue was present in F38, but only on netinst, not Live. In F39, it's now present also on Live. And anything that I build is broken, including F38. But the unofficial respins are not.

Is it possible that this is related to build tools/environment, rather than the included package versions?

Note: When corrupting the images, I started to write the zero byte at 500M mark instead of 200M. This has less likelyhood that I damage files related to early boot, and also the checksum verification counting takes longer, which makes it easier to see what's happening (the display doesn't turn off almost immediately).

[1] https://dl.fedoraproject.org/pub/alt/live-respins/

(In reply to Kamil Páral from comment #19)
> That was simple - just building an image with the latest F38 kernel (and all
> other packages from the main repo) exhibits the bug.
> 
> So we have:
> kernel-6.2.9-300.fc38 - OK
> kernel-6.5.8-200.fc38 - BROKEN
> 
> I guess I'll run through all kernels available in Koji to narrow down the
> scope.

Where you able to narrow this down? I see that there are many simpledrm and sysfb
related changes between those kernel versions, so it would be useful to narrow it
a little bit:

$ git log --pretty=one --abbrev-commit --no-merges v6.2.9..v6.5.8 -- drivers/firmware/sysfb*.c
60aebc955949 drivers/firmware: Move sysfb_init() from device_initcall to subsys_initcall_sync
1b617bc93178 firmware/sysfb: Fix VESA format selection
3615c78673c3 efi: sysfb_efi: Fix DMI quirks not working for simpledrm
f35cd3fa7729 firmware/sysfb: Fix EFI/VESA format selection

$ log --pretty=one --abbrev-commit --no-merges v6.2.9..v6.5.8 -- drivers/gpu/drm/tiny/simpledrm.c
7e34cec74ee1 drm/tiny: correctly print `struct resource *` on error
7fa5047a436b drm: Use of_property_present() for testing DT property presence
2a6d731a8f16 drm/simpledrm: Allow physical width and height configuration via panel node
e566507bf2f4 drm/simpledrm: Fix an NULL vs IS_ERR() bug
9a10c7e6519b drm/simpledrm: Add support for system memory framebuffers
fa904b4cbc60 drm/simpledrm: Use struct iosys_map consistently
d3533a8af484 drm/fb-helper: Replace bpp/depth parameter by color mode
29fca6d56d76 drm/format-helper: Simplify drm_fb_build_fourcc_list()
88f19f8bdc45 drm/simpledrm: Set preferred depth from format of scanout buffer

(In reply to Javier Martinez Canillas from comment #23)
> Where you able to narrow this down? 

Please see my comment 22. Anything I build locally is broken, including kernel-6.2.9-300.fc38 (which is fine on the official F38 Workstation Live, but not fine on F38 Everything netinst). So I suspect this could be related to build tooling (is that possible?), rather than kernel. Or there's some other factor which I currently don't understand.

The unofficial respins are still built with livecd-creator, rather than livemedia-creator, which is definitely a substantial difference. Not sure how it would affect this bug. Can you check what the boot menu options actually *are* in each case?

Could you also grab an F38 live image built by openQA - e.g. https://openqa.fedoraproject.org/tests/2231673/asset/iso/02231673-Fedora-Workstation-Live-x86_64-FEDORA-2023-333b8e1e7f.iso if it's not garbage-collected by the time you read this, if it is, just find the tests of any recent F38 update and grab the ISO from the Workstation live image build test - and see if that's affected? Those are built on an up-to-date F38 toolchain, with livemedia-creator.

(In reply to Kamil Páral from comment #24)
> (In reply to Javier Martinez Canillas from comment #23)
> > Where you able to narrow this down? 
> 
> Please see my comment 22. Anything I build locally is broken, including
> kernel-6.2.9-300.fc38 (which is fine on the official F38 Workstation Live,
> but not fine on F38 Everything netinst). So I suspect this could be related
> to build tooling (is that possible?), rather than kernel. Or there's some
> other factor which I currently don't understand.

Weird. Can you compare what's included in the initrd for both cases ?

(In reply to Adam Williamson from comment #25)
> Could you also grab an F38 live image built by openQA - e.g.
> https://openqa.fedoraproject.org/tests/2231673/asset/iso/02231673-Fedora-
> Workstation-Live-x86_64-FEDORA-2023-333b8e1e7f.iso 

That one works fine, no issues. I also tested one created for F39 [1], and that exhibits the bug.

[1] https://openqa.fedoraproject.org/tests/2230155/asset/iso/02230153-Fedora-Workstation-Live-x86_64-FEDORA-2023-f123156454.iso

(In reply to Adam Williamson from comment #25)
> The unofficial respins are still built with livecd-creator, rather than
> livemedia-creator, which is definitely a substantial difference. Not sure
> how it would affect this bug. Can you check what the boot menu options
> actually *are* in each case?

I also used livecd-creator, but couldn't create an unaffected image. When I have time, I'll try to create it on F38 host instead of F39 host, whether that changes something.

UEFI boot cmdline:
* F38 Workstation Live:
  linuxefi /images/pxeboot/vmlinuz root=live:CDLABEL=Fedora-WS-Live-38-1-6 rd.live.image rd.live.check quiet
* F39 Workstation Live RC 1.2:
  linuxefi /images/pxeboot/vmlinuz root=live:CDLABEL=Fedora-WS-Live-39-1-2 rd.live.image rd.live.check quiet
* F38 unofficial respin:
  linuxefi /images/pxeboot/vmlinuz root=live:CDLABEL=F38-WORK-x86_64 rd.live.image rd.live.check quiet
* My locally built F38 with all updates:
  linuxefi /isolinux/vmlinuz0 root=live:LABEL=f38updates ro rd.live.image quiet rhgb rd.live.check

When building the Live image, I ran:
$ sudo setenforce 0
$ sudo livecd-creator -c live.ks --releasever 38 --cache /var/cache/live -f f38updates

Where live.ks is this:

%include fedora-kickstarts/fedora-live-workstation.ks
%packages
%end
%post
%end

I wonder why my boot cmdline is different (and whether it matters). Also, in BIOS mode, it has the old colorful graphics layout that we haven't used for multiple releases (perhaps it's syslinux instead of grub, or something like that?).


(In reply to Javier Martinez Canillas from comment #26)
> Weird. Can you compare what's included in the initrd for both cases ?

I can try on Monday.

(In reply to Adam Williamson from comment #12)
> dracut specifically turns off the plymouth splash for the duration of the
> media check. this is the dracut code:

I believe I found a lead in that code you pasted. Assuming the DRACUT_SYSTEMD variable is set, the `die` function will halt the system, which unfortunately turns the screen black without powering off / rebooting.

Code shown below (modules.d/99base/dracut-lib.sh):

    if [ -n "$DRACUT_SYSTEMD" ]; then
        systemctl --no-block --force halt
    fi

When I run that `systemctl --no-block --force halt` command as root on Fedora 38, my screen goes black within 1 second (after briefly showing a message "kauditd_printk_skb: X callbacks suppressed").

well, that hasn't changed since 2014 either. so possibly either systemd's behaviour on halt has changed, or the kernel's behaviour on halt has changed. This still doesn't explain the mystery of kparal's results, though. Thinking about it, I believe f38 and f39 currently are on about the same kernel version - 38 is actually a little ahead since 39 is in freeze - so if current F38 openQA lives don't show the problem, it *probably* isn't the kernel.

possibly systemd? I guess I can try building a newer systemd on f38 and testing (or an older systemd on f39).

Adding `rd.systemd.debug_shell` to the kernel command line in GRUB, then switching to tty9 (Ctrl+Alt+F9) while the Media Check on F38 Live is running, gives you a basic debug shell from the initramfs. lsmod shows that the GPU driver (i915 in my case) has not been loaded yet, and manually typing the `systemctl --no-block --force halt` command from here *does not* blank the screen.

OTOH, you can also boot directly to a very minimal environment with the GPU driver loaded but not much else by passing `rd.systemd.debug_shell systemd.mask=systemd-udevd.service plymouth.enable=0 systemd.unit=emergency.target`. In this environment the screen *does* blank for me when I issue the halt command.

So it seems like a boot order thing --> the media check needs to happen before the GPU driver loads (is my theory anyway). But I'm not actually sure which service is actually loading the driver since I masked out systemd-udevd but it still got loaded...

One interesting note: The emergency shell (tty2: Ctrl+Alt+F2) refuses to work on F38 Live since there's no root password, but works on my host F38 machine. From the emergency shell, if you unload modules such as "zram", "loop", and "fuse" using `modprobe --remove` and then exit out of the emergency shell, it actually re-loads those modules again:

    [root@localhost ~]# exit
    exit
    Reloading system manager configuration
    ...
    [  489.741373] zram: Added device: zram0
    ...
    [  490.017024] loop: module loaded
    [  490.035202] fuse: init (API version 7.38)
    ...

I suspect that system manager might be responsible for loading the GPU driver...?

Yep, sure enough on F39 Live, running lsmod from the Debug Shell (Ctrl+Alt+F9) during the media check shows i915 *is* loaded at this time, and issuing the halt command *does* blank the screen.

It's also pretty obvious on my laptop whenever the GPU driver loads since the screen goes blank for about 1/4 of a second, then shows a bunch of i915 messages. On F38 Live this happens after the media check; but on F39 Live it happens before the media check.

thanks for that detective work! this does make it sound quite a lot like a kernel thing...though in that case it's odd that updated F38 images work OK.

(In reply to Adam Williamson from comment #32)
> thanks for that detective work! this does make it sound quite a lot like a
> kernel thing...though in that case it's odd that updated F38 images work OK.

I did some more digging. I extracted the initrd of the ISO files:

    $ mkdir f39
    $ sudo mount -o loop,ro Fedora-Workstation-Live-x86_64-39_Beta-1.1.iso f39
    $ cp f39/images/pxeboot/initrd.img initrd.img
    $ sudo umount f39
    $ xz -d -k < initrd.img > initrd.cpio
    $ mkdir extracted
    $ cd extracted
    $ cpio -idmv < ../initrd.cpio

F39 Live puts i915 and amdgpu modules and firmwares in the initrd:
* /usr/lib/firmware/i915/
* /usr/lib/firmware/amdgpu/
* /usr/lib/modules/6.5.2-301.fc39.x86_64/kernel/drivers/gpu/drm/i915/i915.ko.xz
* /usr/lib/modules/6.5.2-301.fc39.x86_64/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko.xz

F38 Live does not have those files in the initrd.

F39 Live also has a line "drm" in usr/lib/dracut/modules.txt (in initrd). F38 Live does not have that line.

So I believe F39 Live is loading the GPU driver super early, from some initrd service (that gets run before the media check), and setting up a framebuffer for the tty; then when dracut issues a system halt, the framebuffer dies and the screen blanks. The screen didn't blank on F38 Live since the GPU/framebuffer doesn't get set up until after the squashfs gets mounted (since that's where the .ko.xz files are located on F38 Live).

Curiously enough, systemd distinguishes "initrd services" and "main system services" in a couple of its kernel flags (systemd.mask vs. rd.systemd.mask, systemd.wants vs. rd.systemd.wants) which you can read about here: https://www.freedesktop.org/software/systemd/man/latest/systemd-debug-generator.html

F38 Netinstall also has i915/amdgpu files in the initrd and has "drm" in the /usr/lib/dracut/modules.txt file (in the initrd).

Thanks again - really great job on figuring that out.

so, I can actually contribute something useful at this point :D

comparing logs of an f38 live image build with logs of an f39 live image build (both in openQA), here's a key difference:

F38 - 2023-10-27 15:25:24,130 INFO pylorax: dracut args = ['--xz', '--add', 'livenet dmsquash-live dmsquash-live-ntfs convertfs pollcdrom qemu qemu-net', '--omit', 'plymouth', '--no-hostonly', '--debug', '--no-early-microcode']
F39 - 2023-10-27 14:52:01,497 INFO pylorax: dracut args = ['--xz', '--add', 'livenet dmsquash-live dmsquash-live-ntfs convertfs pollcdrom qemu qemu-net', '--no-hostonly', '--debug', '--no-early-microcode']

note we've lost "--omit plymouth". plymouth depends on drm. So, that's almost certainly the key change here. And here's where it happened:
https://github.com/weldr/lorax/commit/386545a5281fab4a4440fb60513429ef2a866e58

so ultimately a lorax change triggered this. Probably the easiest thing to do is just revert that, though it may not be the *right* thing to do - the *right* thing may be to figure out a way to keep the message showing even if the DRM driver is in the initramfs...

so this neatly explains kparal's differing results too, as livecd-creator never omitted plymouth.

thinking it through, though...this leaves us with a bit of a conundrum. if we just go back to leaving plymouth out of the live images, that doesn't fix it for installer images, so that doesn't really seem like the right fix. but then what *is* the right fix, and can we do it very quickly (since we need a new RC soon)?

A bodge I can think of is to have dracut print a failure message then wait some time before halting or powering down the system. That might even be "the right fix", in the end. It would at least work consistently across lives and installer images. I'll throw together a test patch for that and try it out.

So I also discovered why the halt was added in the first place: it's necessary for FIPS compliance on at least one failure path - https://bugzilla.redhat.com/show_bug.cgi?id=1053655 .

It seems to me that there shouldn't be any FIPS or security issue with *not* halting when a media check fails, so maybe we can do that. Sure, that means you could maybe break out of the failure and continue booting the system, but that feels to me like a less-bad problem than "you can't see the failure message". A normal person will just see the failure message and reboot. If you for some reason decide to try and l33t h4x0r your way out of the failure, that's on you.

I'm testing a patch that should make that happen right now.

just as an update here - it's turning out to be weirdly hard to get this to stop halting. just attempting to avoid the explicit `systemctl --no-block --force halt` doesn't seem to be enough; I think we wind up running into the emergency halt in dracut-emergency.sh , or something, really can't be sure. I'm trying yet another variation ATM. If anyone else can figure this out that'd be great.

We can fall back to the "print a failure message then wait some time before halting or powering down the system" plan if all else fails, I guess - that one *should* be easier to implement as it basically just involves slapping a sleep 60 in there - but I don't like that plan as much as I guess it's fairly common to just start the media check then go look at something else, and you could easily miss seeing the message in the time it's shown. We could make it a sleep 600...or something...

(In reply to Adam Williamson from comment #38)
> just as an update here - it's turning out to be weirdly hard to get this to
> stop halting. just attempting to avoid the explicit `systemctl --no-block
> --force halt` doesn't seem to be enough; I think we wind up running into the
> emergency halt in dracut-emergency.sh , or something, really can't be sure.
> I'm trying yet another variation ATM. If anyone else can figure this out
> that'd be great.
> 
> We can fall back to the "print a failure message then wait some time before
> halting or powering down the system" plan if all else fails, I guess - that
> one *should* be easier to implement as it basically just involves slapping a
> sleep 60 in there - but I don't like that plan as much as I guess it's
> fairly common to just start the media check then go look at something else,
> and you could easily miss seeing the message in the time it's shown. We
> could make it a sleep 600...or something...

Could also make it say "Press return to reboot" or something along those lines, then issue a reboot instead of a halt -- which I like since you don't have to hold the power button for 5 seconds.

Just gotta test it to make sure the user can't bypass the prompt, or cause the message to disappear, via Ctrl+C/Ctrl+Z or unplugging the keyboard or serial port (SIGINT, SIGHUP and such). And it would probably be a smart idea to test on at least one embedded system like a Raspberry Pi to make sure the prompt doesn't simply immediately disappear due to stdin being unavailable or something.

FYI: systemd since June 2023 has a couple new `SuccessAction/FailureAction` settings for services: `(halt|halt-force|halt-immediate)`. I tried the latter one out on F39 Live (created a custom systemd service with `FailureAction=halt-immediate` and then started the service), but it too blanked my screen. ¯\_(ツ)_/¯

ugh. it's super weird - even making the media check code this:

    if [ $? -eq 1 ]; then
        {
            echo "<24>dracut: FATAL: Media check failed!"
            echo "<24>dracut: Press any key to reboot"
        } > /dev/kmsg
        read anykey
        if [ -n "$DRACUT_SYSTEMD" ]; then
            systemctl --no-block --force reboot
        else
            reboot
        fi
    fi

doesn't work, somehow. The message is printed...and then it behaves exactly the same, the screen blanks, the system doesn't reboot. even if somehow `read` doesn't work here, it should reboot, not halt. No idea why it's doing that.

I'm trying the 'sleep for a long time' option now in lieu of getting anything better to work :(

GAH. I've just realized, I've been hoist by my own petard: my own clever don't-keep-redownloading-the-same-update code in the openQA scheduler (I'm using openQA to build images for testing) has been reusing the same old patched dracut over and over again, I think. siiigh. will have to twiddle a bit and restart testing earlier versions I thought were better but "failed testing".

No, wait, that's wrong. It caches by task ID. I just checked and the image for my last scratch run does seem to have been patched correctly. But what it prints on the console doesn't match what I patched in, and it still blanks. sigh. this is really biting my shiny metal ass.

gahh. I just found there's a different path with the same "CD check failed!" error message, in a script called anaconda-diskroot that's part of the anaconda package. Possibly we run into *that* on netinsts, not the one in dmsquash-live-root.sh...

Okay, so I'm *hoping* that when https://openqa.stg.fedoraproject.org/tests/3302257#live and https://openqa.stg.fedoraproject.org/tests/3302250#live complete, the images they build will have this fixed: they will print a message and pause for 12 hours, then halt. If that works, I'll send official builds. Somebody else might be able to come up with a better idea, but this is the best I can work out on a short time frame.

FEDORA-2023-40de14d7fc has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-40de14d7fc

FEDORA-2023-40de14d7fc has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-40de14d7fc`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-40de14d7fc

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

(In reply to Fedora Update System from comment #45)
> FEDORA-2023-40de14d7fc has been submitted as an update to Fedora 39.
> https://bodhi.fedoraproject.org/updates/FEDORA-2023-40de14d7fc

I built Workstation Live and Everything netinst locally with these updates, the problem is gone, media check failure stays on screen. Good job!
I also performed an installation in VMs, works fine. So this seems resolved. But let's keep this open until we can verify it with the next RC as well.

If I wasn't a complete idiot, I would've first checked whether there was a new RC already, before building my own images. Oh well.
Yes, it works fine with RC1.4 as well. Setting as verified.

FEDORA-2023-40de14d7fc has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.