Image I tested with: https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20171128.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-netinst-x86_64-Rawhide-20171128.n.0.iso

Still the case with Fedora-Rawhide-20171211.n.0.

what if you add "rhgb" to the kernel command line ? does it work then?

Hey, good call! Yes, it does. 'rhgb' is not on the installer cmdline by default, and adding it does result in a successful boot to the graphical installer.

so there's a plymouth bug with text mode fallback plugin I guess, but also a problem with the lorax config ? Not sure how to make sure rhgb gets added the kernel command line again. bcl do you know?

Well, it seems that rhgb being in the installer cmdline isn't new at all; I tried a few images from earlier releases, and it wasn't there at least as far back as F20. It may in fact be intentional - I suspect we might have decided at some point that we don't *want* graphical boot for installer images.

Sigh - I meant "rhgb **NOT** being in the installer cmdline". I haven't yet found a release which *did* have it in the cmdline for installer images.

okay, well we definitely want graphical boot on install images. showing boot spew and kernel debug messages as the first thing someone sees when they want to try fedora out is not a good idea!

As far as I can see from the lorax commits we have never had rhgb on the cmdline (I'm going to guess that's because it would break on some subset of systems). I'm fine with adding it as long as the Anaconda team is ok with it.

To add it all of the config files under: https://github.com/rhinstaller/lorax/tree/master/share/templates.d/99-generic/config_files need to have it added. I'm not sure which arches it is appropriate for. s390? etc.

So, where are we going with this? bcl, are you waiting on someone to sign off on it, or submit a PR, or what? Thanks!

Yeah, I'd like to get the anaconda team's input here before I make any changes.

In that case we'd better CC them =)

This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle.
Changing version to '28'.

Ping? Anyone?

...still ping? I'm still stuck using three different video drivers in three different openQA test configs and encountering different bugs all over the place, it's getting annoying, and this is one of the causes...

Is this about adding rhgb always unconditionally to the kernel boot command line ? Also why exactly is it needed now on UEFI instead of kernel resolving-this automatically (eq. aren't we just papering over a bug in a different component) ?

Supposed we do this, these would be possible issue/questions that come to me mind:
- what if the installation environment fails to start, will users see an appropriate error message or would they see a frozen graphical boot screen forever ?
- I don't think anyone really tested a graphical boot screen for the installation images,  would it be possible to run some automated tests with the boot option added to see if it leads to a spectacular explosion or works fine ?
- is it safe to introduce the option on all platforms uncoditionally, eq. could it cause some mayhem rather than just being ignored on platforms that don't support graphical output ?
- if you request text installation with boot options or kickstart there could be a weird effect where you get a graphical boot screen only to be replaced by the Anaconda text interface (not that big of an issue though)

"Is this about adding rhgb always unconditionally to the kernel boot command line ?"

That's the proposal, yes.

"Also why exactly is it needed now on UEFI?"

Well, the closest we got to determining that was around #c5: "so there's a plymouth bug with text mode fallback plugin I guess". But after that Ray seemed more interested in suggesting that rhgb should always be used for installer boot than in trying to fix that issue.

Your questions all seem like sensible ones to me. I will try to build a modified image with 'rhgb' added to the boot params and run that through staging openQA soon if I can.

I suppose the other thing we can do is just go back again and try to find the bug in plymouth; I did at least narrow it down to having been introduced somewhere between 0.9.3-0.9.20160620git0e65b86c.fc27 and 0.9.3-1.fc28 .

to be clear, we should fix both problems.  we definitely want graphical boot for the installer, and we definitely want text mode plymouth splashes to work

OK, I think I've bisected down to the single commit that broke this in plymouth - it's e4f86e3c , "renderer: export device name from plugin". An image built with a plymouth RPM at that commit fails, an image built with a plymouth RPM at the previous commit (fdda9af2) succeeds.

hey thanks for bisecting.

I don't have time right now to deep dive, but this work-in-progress commit might fix the problem:

https://cgit.freedesktop.org/plymouth/commit/?h=wip/fix-text-splash&id=843b2fd8dc0b13bc84eeff503e68400c4876e19a

OK, will test that.

Note I tried booting with plymouth:debug and, while I can't see *all* the messages as I can't get at the scrollback - I haven't figured out how to redirect the boot messages to a serial console without affecting anaconda's behaviour and making the bug go away - it ends with:

ply_event_loop_handle_disconnect_for_source:calling disconnected_handler 0x55aef7c19ba0 for fd11
ply_event_loop_handle_disconnect_for_source:done calling disconnected_handler 0x55aef7c19ba0 for fd11
ply_event_loop_Free_destinations_for_source:freeing destination (1, 0x55aef7c1a070, 0x55aef7c19ba0) of fd 11
ply_even_loop_remove_source_node:failed to delete fd 11 from epoll watch list: Bad file descriptor

Thanks! Unfortunately that patch doesn't seem to help (after fixing the obvious typo in it, s/render/renderer/).

try plymouth.debug=stream:/dev/ttyS0  to get debug messages to go to serial console

(with no console= lines)

OK, will try that once I'm done with this eternal phone queue to talk to my bank...

Created attachment 1446774 [details]
plymouth debugging output from boot attempt with wip patch

Created attachment 1446787 [details]
more complete plymouth debug log

plymouth-0.9.3-7.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2e38d03103

Sent out an F28 update just so any rebuilt installer images will not have the bug, but the main thing is I already sent -7 to Rawhide, so Rawhide composes after 20180601.n.1 should not have the bug. Thanks Ray!

plymouth-0.9.3-7.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-2e38d03103

Some notes here:

we figured out that this bug happens when the 'frame-buffer' renderer is present but the 'drm' renderer isn't. With plymouth-0.9.3-6 and earlier, the 'frame-buffer' renderer is in the core package but the 'drm' renderer is in the graphics-libs subpackage (since pbrobinson moved it there back in 2015). Ray says it's really a bug in plymouth - it should cope more gracefully with all the possible cases of some renderers being present and others not - but he can't fix it immediately.

So our initial workaround for this, in -7, was to move frame-buffer into graphics-libs alongside drm, so either both are present or neither is. This turned out to be a bad idea, as it turns out there are problems on boot if 'rhgb' is on the kernel command line but the 'frame-buffer' renderer isn't present - which can now happen, since it's not in the core plymouth package any more. In the worst case, if you did a minimal encrypted install, you couldn't boot it properly as you couldn't enter the encryption passphrase on boot.

So I unpushed the F28 update while we chewed on this, and for Rawhide, sent a -8 with the 'drm' and 'frame-buffer' renderers both moved back to the main package, the way things were before Pete's change in 2015. This should hopefully mean neither bug happens any longer, but unfortunately it adds at least one package to a minimal install, 'libdrm', as whichever package the 'drm' renderer is in depends on libdrm. This is why Pete moved the 'drm' renderer into the sub-package in the first place (to avoid that dependency for minimal installs), and he understandably isn't happy with the dep getting moved back to the main package.

We're still chewing over what's the best thing to do for now. Obviously it'd be ideal if Ray could fix up plymouth so it behaves reasonably in all cases of renderers being available or not available, so we can go back to the -6 or -7 state without buggy behaviour.

If we can't fix plymouth itself quickly, another approach would be to go back to the -6 split ('frame-buffer' in the main package, 'drm' in graphics-libs) and pull graphics-libs into the installer environment...

Bit more detail for anyone playing along at home (halfline already had this figured out two days ago but his IRC breadcrumbs were cryptic to me till I figured it out myself :>):

When 'rhgb' isn't on the cmdline, we go down a particular path in plymouth main.c:

        if (!plymouth_should_show_default_splash (&state)) {
                /* don't bother listening for udev events if we're forcing details */
                device_manager_flags |= PLY_DEVICE_MANAGER_FLAGS_IGNORE_UDEV;

the 'if' there is satisfied when 'rhgb' is not on the cmdline, so we get the flag PLY_DEVICE_MANAGER_FLAGS_IGNORE_UDEV set. The commit that introduced this logic is:

https://cgit.freedesktop.org/plymouth/commit/?id=382305e4a9f7b4c221968fcba7394b1cc03b454e

"main: disable hotplug events and splash delay if details forced

There's no point in waiting for a graphics device if details are
forced, and we shouldn't ever delay showing details.  If details
are requested, we shouldn't be hiding them."

Basically, what it does is tell the plymouth device manager to not bother hooking into udev and looking out for graphics device-related events and doing stuff in response to them showing up and disappearing. That sounds peachy and sorta makes sense, except that there's a fallback path too. When it's told to ignore udev, the device manager calls a function 'create_fallback_devices', which is *not* hit when it's using udev. 'create_fallback_devices' then calls 'create_devices_for_terminal_and_renderer_type' with the renderer type set to 'PLY_RENDERER_TYPE_AUTO'...which basically results in plymouth trying each of the x11, drm, and frame-buffer renderers (in order) until it hits one that works.

As we noted, in the installer environment (before we started trying to fix this by moving things around in the packages), the drm and x11 renderer plugins are not present, so those obviously fail...but the frame-buffer renderer *is* present, so plymouth tries it, and finds that /dev/fb0 is present, so it goes ahead and loads it.

Unfortunately /dev/fb0 at this point is backed by the kernel EFI framebuffer driver ('efifb'). The kernel expects, during boot, to switch from efifb to another driver (bochsdrmfb)...but because plymouth is holding the device, it cannot. Hence why the boot hangs at "fb: switching to bochsdrmfb from EFI VGA".

One outstanding mystery for me at least is why this only fails (apparently) with 'vga' / 'std' but works with other drivers, because it seems like the same stuff happens there. From the plymouth debug logs plymouth still loads the frame-buffer driver and uses /dev/fb0...but somehow, the kernel doesn't get stuck trying to unregister efifb when the new driver is qxldrmfb rather than bochsdrmfb. I'm really not sure why not.

I'm sort of wondering if it maybe would make more sense to try and send the device manager down a path where it just goes straight to PLY_RENDERER_TYPE_NONE, rather than PLY_RENDERER_TYPE_AUTO, for this specific codepath. That seems to fit the initial intention of not bothering with a 'graphics device' a bit better?

Created attachment 1448545 [details]
proposed patch for plymouth

attachment 1448545 [details] is upstream now:

https://cgit.freedesktop.org/plymouth/commit/?id=014c2158898067176738ec36c9c90cc266a7e35b

plymouth-0.9.3-9.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2e38d03103

For the record, -9 has this setup:

* https://cgit.freedesktop.org/plymouth/commit/?id=014c2158898067176738ec36c9c90cc266a7e35b backported - it actually fixes the initial bug here (UEFI boot with 'std' / 'vga', frame-buffer present but drm not)
* frame-buffer and drm renderers both moved back to the graphics-lib subpackage, because we think that's ultimately the 'most correct' layout if we want to avoid a libdrm dep in the main package
* https://cgit.freedesktop.org/plymouth/commit/?id=bdfcf889f8cda47190d98fa8a3e401a1db38074c backported - it fixes the bug that showed up when we initially tried moving frame-buffer to the subpackage, that booting with 'rhgb' on the cmdline but no renderers installed gets stuck if an encryption passphrase is needed

basically, it should hopefully fix everything and lead us all to a glorious land of honey. So please tell me when it blows up your system instead...

plymouth-0.9.3-9.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-2e38d03103

plymouth-0.9.3-9.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.