Bug 1706557

Summary: Lost plymouth boot splash after upgrade from 29 to 30
Product: [Fedora] Fedora Reporter: edward <edwardp>
Component: kernelAssignee: Hans de Goede <hdegoede>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 30CC: airlied, bskeggs, edwardp, elad, gnome-sig, hdegoede, ichavero, itamar, jarodwilson, jeremy, jglisse, john.j5live, jonathan, josef, kernel-maint, linville, masami256, mchehab, mjg59, rstrode, steved
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-04 12:00:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Output of plymouth.log
none
Screenshot of monitor before desktop manager/login prompt appears
none
plymouth-charge.log
none
plymouth-spinner.log
none
dmesg.charge
none
dmesg.spinner none

Description edward 2019-05-05 12:02:13 UTC
Description of problem:

Upgraded system from Fedora 29 to 30, upon first boot into upgraded system, the pllymouth boot splash (set to spinfinity) does not appear. The screen starts as something akin to a terminal window and lists various processes before it stops, then the GUI desktop manager and login prompt appears.

After redoing the command 'sudo plymouth-set-default-theme spinfinity -R' and rebooting, the above continues to occur.

Video is NVIDIA using the nouveau driver:

00:0d.0 VGA compatible controller: NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) (prog-if 00 [VGA controller])
	Subsystem: Hewlett-Packard Company Device 2a6c
	Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 23, NUMA node 0
	Memory at fb000000 (32-bit, non-prefetchable) [size=16M]
	Memory at e0000000 (64-bit, prefetchable) [size=256M]
	Memory at fc000000 (64-bit, non-prefetchable) [size=16M]
	[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: [48] Power Management version 2
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
	Kernel driver in use: nouveau
	Kernel modules: nouveau

Comment 1 Hans de Goede 2019-06-12 09:17:11 UTC
This sounds like your system is not configured to use plymouth during boot.

What is the output of:

cat /proc/cmdline

?

Comment 2 edward 2019-06-12 21:38:34 UTC
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.1.6-300.fc30.x86_64 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet

Comment 3 edward 2019-06-12 21:59:44 UTC
sudo systemctl status plymouth-start.service
● plymouth-start.service - Show Plymouth Boot Screen
   Loaded: loaded (/usr/lib/systemd/system/plymouth-start.service; static; vendor preset: disabled)
   Active: inactive (dead) since Wed 2019-06-12 17:50:38 EDT; 7min ago
  Process: 539 ExecStartPost=/usr/bin/plymouth show-splash (code=exited, status=0/SUCCESS)
 Main PID: 538 (code=exited, status=0/SUCCESS)

Jun 12 17:49:44 downstairs systemd[1]: Starting Show Plymouth Boot Screen...
Jun 12 17:50:38 downstairs systemd[1]: plymouth-start.service: Succeeded.

Comment 4 Hans de Goede 2019-06-13 08:39:55 UTC
Ok.

Can you please add "plymouth.debug=stream:/run/plymouth.log" to your kernel commandline. Reboot and the after booting attach the generated /run/plymouth.log file here please?

Comment 5 edward 2019-06-13 21:57:49 UTC
I am getting a 500 internal server error when trying to attach the file. A chrome-based browser displayed a different error indicating the site was unavailable - access denied.

Comment 6 edward 2019-06-13 22:05:35 UTC
Created attachment 1580406 [details]
Output of plymouth.log

The system took some extra time to boot up once the line was added, there were several seconds of no hard drive activity, before it resumed.

The spinfinity bootsplash screen eventually displayed, however, the progress bar at the bottom had already reached the opposite end (completion) by the time it displayed.

Comment 7 edward 2019-06-13 22:10:56 UTC
# plymouth-set-default-theme --list
bgrt
charge
details
spinfinity
spinner
text
tribar

Comment 8 Hans de Goede 2019-09-07 21:19:27 UTC
Thank you for collecting the log, and sorry for being so slow to respond. This one somehow got of my radar.

I've just prepared an updated plymouth package which may also help with this issue:
https://bodhi.fedoraproject.org/updates/FEDORA-2019-aadaa1a995

Please give this version a try.

Comment 9 edward 2019-09-19 18:35:47 UTC
Unfortunately, this update to the Plymouth packages, did not resolve the problem. The bootsplash still does not appear.

I am attaching a screenshot of the monitor, showing (mostly) what appears on screen before the desktop manager/login prompt appears.

Comment 10 edward 2019-09-19 18:36:30 UTC
Created attachment 1616859 [details]
Screenshot of monitor before desktop manager/login prompt appears

Comment 11 Hans de Goede 2019-09-20 13:17:07 UTC
I've checked you log file again and I'm afraid I have no idea why things do not work.

Comment 12 edward 2019-09-22 23:08:29 UTC
Is it possible that "spinfinity" might not be compatible with this particular Nvidia graphics chip? 

I just changed the bootsplash theme from "spinfinity" to "charge", then rebooted. Upon each subsequent reboot, "charge" appears as expected.

Comment 13 Hans de Goede 2019-09-23 09:17:01 UTC
Changing from "spinfinity" to "charge" should not make a difference, but when you did so I assume you re-generated your initrd to actualy apply the change?

Re-generating the initrd may have helped, perhaps you did not re-generate it after installing the latest plymouth update, or perhaps some other component (kernel or linux-firmare) was updates which helps.

Can you try changing the theme back to "spinfinity" again? I think that you will find that that works fine too when using a freshly generated initrd.

Comment 14 edward 2019-09-23 22:41:23 UTC
I did not manually regenrate initrd when the plymouth packages were updated, unless that process was automated during the update.

Yes, it was regenerated when I changed it from "spinfinity" to "charge". 

I also regenerated it when switching it from "charge", back to "spinfinity". Upon a reboot after this change, the same text screen as shown in the attachment to Comment 10 was displayed. So there seems to be something different about "spinfinity". 

As of now, I have not tried any of the other bootsplash themes.

Comment 15 Hans de Goede 2019-09-24 07:32:29 UTC
Hmm, you might be right and this might be theme related, I was confusing spinfinity and the spinner, spinner and charge both use the two-step splash plugin whereas spinfinity uses the throbgress plugin.

I will try to reproduce this by using the spinfinity theme myself and then we will see from there. Note this is going to be somewhat lower on my priority list, so I'm not sure when I'm going to get around to doing this. In the mean time you may want to give the "spinner" theme a shot, this is somewhat like spinfinity.

Comment 16 edward 2019-09-24 23:33:54 UTC
I have the following themes installed on this system:

bgrt
charge
details
fade-in
script
spinfinity
spinner
text
tribar

Out of all of these, only "charge", "details", "fade-in", "text" and "tribar" worked.

"bgrt" and "spinner" (as is the case with "spinfinity") displayed the same text scrolling attached to Comment 10, both stopping at the same point. 

"details" displayed the same text scrolling, except it continued all the way through to the end, then the login screen appeared. I therefore consider "details" as working.

"script" displayed nothing but a black screen.

Comment 17 Hans de Goede 2019-09-25 11:10:12 UTC
Hmm, bgrt, spinner and charge are all using the same code underneath, so this is weird.

Can you collect a plymouth.log as before from both a boot with charge and from a boot with spinner? And please also collect dmesg output of both boots, by doing:

dmesg > dmesg.spinner

resp.

dmesg > dmesg.charge

Once the system is booted.

After doing that please try the following:


As root edit /etc/plymouth/plymouthd.conf and add the following:

[Daemon]
DeviceTimeout=60

And then set the theme to say spinner (and regenerate the initrd) and see if that helps?

Comment 18 edward 2019-09-25 22:09:08 UTC
In changing it back to spinner, with the plymouth.debug command in the boot string, spinner briefly appeared somewhere in between the middle and end of the bootup, but not before a text screen. Without that command, the same text screen per Comment 10 appeared instead. It otherwise will briefly appear after logging out and before the reboot.

Appending that one line to plymouthd.conf had no affect, the text screen still appeared when set to spinner.

Comment 19 edward 2019-09-25 22:43:03 UTC
This site can’t be reached
The webpage at https://bugzilla.redhat.com/attachment.cgi might be temporarily down or it may have moved permanently to a new web address.
ERR_ACCESS_DENIED

Unable to upload files to the bug report. I will e-mail them to you.

Comment 20 edward 2019-09-25 22:49:50 UTC
There is a permissions issue with both of the log files, I am not able to upload them anywhere, nor in e-mail. They

Comment 21 edward 2019-09-25 22:53:20 UTC
Created attachment 1619285 [details]
plymouth-charge.log

Comment 22 edward 2019-09-25 22:53:43 UTC
Created attachment 1619286 [details]
plymouth-spinner.log

Comment 23 edward 2019-09-25 22:54:01 UTC
Created attachment 1619287 [details]
dmesg.charge

Comment 24 edward 2019-09-25 22:54:18 UTC
Created attachment 1619288 [details]
dmesg.spinner

Comment 25 Hans de Goede 2019-10-21 18:03:59 UTC
Hi Edward,

Thank you for the info.

I see it has taken me almost a month to get back to you, sorry about that.

I'm a bit closer to figuring out what is going on. It seems that when using the spinner theme, the plymouthd instance inside the initrd crashes (or does not run) and then plymouthd gets started again after the switch-root.

Can you try the following:

1. 'sudo plymouth-set-default-theme spinner -R'
2. 'sudo mv /usr/sbin/plymouthd /usr/sbin/plymouthd.bak'
3. Reboot with "plymouth.debug=stream:/run/plymouth.log" on you kernel commandline.
4. Collect /run/plymouth.log and attach it here ?

The idea is to avoid plymouthd (re)running after the switchroot from the initrd to the actual root, so that it will not overwrite the /run/plymouth.log written by the plymouth inside the initrd and then I can hopefully get something useful out of the log from the initrd instance.

Regards,

Hans

Comment 26 Hans de Goede 2019-10-21 21:01:26 UTC
Good news, I've been digging through my stack of old GPUs, I've found one similar to the one integrated into your motherboard and I've managed to (probably) reproduce your problem and I'm homing in on a fix.

Comment 27 Hans de Goede 2019-10-24 10:43:42 UTC
So it seems that this is a kernel bug and I've written a nouveau patch which should fix this.

I've started a scratch kernel build with the patches which should fix this here:
https://koji.fedoraproject.org/koji/taskinfo?taskID=38517104

Note this is still building atm, this takes a couple of hours.

Here are some generic instructions for installing a kernel directly from koji:
https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

If this does not fix the problem, please collect a new plymouth-debug log file.


p.s.

Your chipset integrated gpu is a nv4x Nvidia chip, on the NV43 GPU I use for testing the screen will not wake up after going to sleep. A workaround for me is to do CTRL+ALT+F6 to switch to a text console and the switch back to ALT+F2 are you seeing this too?

Comment 28 Hans de Goede 2019-10-24 10:44:51 UTC
Note that koji removes the results of scratch / test-builds after a couple of days to free up diskspace. So if you do not have time to test this soon, please at least download the rpms (see the instructions for which one you need).

Comment 29 edward 2019-10-25 22:24:58 UTC
It looks like it is no longer available for download. 

Although the 5.3.7 kernel is now available, I looked at https://koji.fedoraproject.org/koji/buildinfo?buildID=1402639 and it shows the build date as Friday October 18. I'm presuming your fix is not in this released kernel. Is there a way to check to see if the fix will be included with the next version (5.3.8?)?

Comment 30 Hans de Goede 2019-10-26 12:08:00 UTC
Koji is not that quick with recycling scratch builds, you need to follow the x86_64 build link to get the downloads:
https://koji.fedoraproject.org/koji/taskinfo?taskID=38517146

And yes the regular builds do not contain the patches, you really need the scratch build.

Comment 31 edward 2019-10-26 23:54:32 UTC
$ sudo rpm -ivh --oldpackage kernel*.rpm

error: Failed dependencies:
        kernel-core-uname-r = 5.3.7-200.rhbz1706557.fc30.x86_64 is needed by kernel-5.3.7-200.rhbz1706557.fc30.x86_64
        kernel-uname-r = 5.3.7-200.rhbz1706557.fc30.x86_64 is needed by kernel-modules-5.3.7-200.rhbz1706557.fc30.x86_64

Comment 32 edward 2019-10-27 00:08:54 UTC
Sorry about that, downloaded kernel instead of kernel-core.

With the kernel build from the above link, the bug is fixed.

My questions are: 

Will this patch be incorporated into an upcoming new version of the kernel? 

And can the scratch build be uninstalled easily, or will it be removed automatically when the next three regular kernels are eventually installed?

Comment 33 edward 2019-10-27 14:05:43 UTC
With this scratch build still installed, the system would not update packages. The errors received were duplicates of the errors in comment 31, except they referenced kernel version 5.3.5. 5.3.6 was the latest kernel installed prior to installing the scratch build.

I rebooted into 5.3.6 and used "sudo rpm -ev --nodeps {kernel-core...}.rpm" to remove the scratch build and in the process it also appears to have automatically removed the associated kernel-modules package and update grub. After rebooting into 5.3.6 again, the system was able to update. My second question has been answered.

Comment 34 Hans de Goede 2019-10-29 11:44:21 UTC
(In reply to edward from comment #32)
> With the kernel build from the above link, the bug is fixed.

Thanks, that is good news, that means that this indeed is a nouveau bug as I expected.

> Will this patch be incorporated into an upcoming new version of the kernel? 

I have submitted the patch to the mainline kernel developers, once it is accepted there, I can add it as a downstream patch to the Fedora kernel packages and then once Fedora switches to a new enough upstream / mainline kernel it can be dropped as that version will include it.

For now we need to wait a but for upstream review of the patch to happen first.

Comment 35 edward 2019-10-30 00:09:37 UTC
Fedora 31 was released today. Should I wait until the patch is in a succeeding Fedora 30 kernel first, before upgrading to 31, or would be it OK to upgrade it to 31 now?

Comment 36 Hans de Goede 2019-10-30 08:43:32 UTC
(In reply to edward from comment #35)
> Fedora 31 was released today. Should I wait until the patch is in a
> succeeding Fedora 30 kernel first, before upgrading to 31, or would be it OK
> to upgrade it to 31 now?

That is a good question, if you use the official update method then plymouth is used to show progress during the update, so then things will likely fail I think. Or they might work but you will not see progress.

You can remove rhgb from your kernel commandline, then you will get a textmode boot and then the upgrade _should_ work, or you can wait for the bug to be fixed in F30 first.

Comment 37 edward 2019-12-02 12:21:54 UTC
Hans,

Would you know how long it takes for a patch to be incorporated into the kernel? 

5.3.13-200 came in for Fedora 30 this morning and apparently, the patch has yet to make it into the kernel. I have had the default bootsplash set to spinfinity and am still getting the text boot.

Comment 38 Hans de Goede 2019-12-17 14:36:32 UTC
Good news the patches fixing this have been merged upstream in 5.5-rc2. I have backported them to the Fedora 5.4 kernel series which is currently being prepared and should show up in updates-testing soon-ish.

In the mean time I've started a scratch-build of 5.4 with the patches added here:
https://koji.fedoraproject.org/koji/taskinfo?taskID=39689689

So if you want, you can use that version until 5.4 hits updates-testing.

Comment 39 Hans de Goede 2020-03-04 12:00:00 UTC
The stable updates repository has 5.5.7 know which includes the kernel patches fixing this, closing.

Comment 40 edward 2020-03-04 12:11:41 UTC
Hello Hans.

Thank you for fixing the bug. I can confirm the boot splash now working in both the 5.5 kernel and your back porting the fix to the 5.4 kernel in both Fedora 31 and 30.