Bug 1683197 - gdm Fails to load with "nomodeset" [NEEDINFO]
Summary: gdm Fails to load with "nomodeset"
Status: ASSIGNED
Alias: None
Product: Fedora
Classification: Fedora
Component: xorg-x11-drv-qxl   
(Show other bugs)
Version: 30
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Alon Levy
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: BetaFreezeException, F30BetaFreezeException F30FinalBlocker, FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2019-02-26 12:42 UTC by František Zatloukal
Modified: 2019-03-22 19:35 UTC (History)
31 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
awilliam: needinfo? (ajax)


Attachments (Terms of Use)
Xorg log with QXL virtual hardware (bad) (8.07 KB, text/plain)
2019-03-19 11:44 UTC, Zbigniew Jędrzejewski-Szmek
no flags Details
Xorg log with QXL virtual hardware with vesa driver uninstalled (bad) (8.26 KB, text/plain)
2019-03-19 11:45 UTC, Zbigniew Jędrzejewski-Szmek
no flags Details
Xorg log with QXL virtual hardware with vesa and qxl drivers uninstalled (good) (21.89 KB, text/plain)
2019-03-19 11:46 UTC, Zbigniew Jędrzejewski-Szmek
no flags Details
Xorg log with virtio virtual hardware (good) (22.10 KB, text/plain)
2019-03-19 11:46 UTC, Zbigniew Jędrzejewski-Szmek
no flags Details
Xorg log with VGA virtual hardware (bad) (8.07 KB, text/plain)
2019-03-19 11:46 UTC, Zbigniew Jędrzejewski-Szmek
no flags Details
Xorg log with VGA virtual hardware with qxl driver uninstalled (bad) (8.08 KB, text/plain)
2019-03-19 11:47 UTC, Zbigniew Jędrzejewski-Szmek
no flags Details
Xorg log with VGA virtual hardware with qxl and vesa drivers uninstalled (good) (21.58 KB, text/plain)
2019-03-19 11:47 UTC, Zbigniew Jędrzejewski-Szmek
no flags Details

Description František Zatloukal 2019-02-26 12:42:24 UTC
Description of problem:
Whenever is "nomodeset" used as a boot parameter, the boot process hangs right when starting gdm.

It doesn't make a difference if I force gdm to use Xorg. I am able to reproduce this in VM (both QXL and virtio) and bare metal (RadeonSI driver / AMD gpu).

Version-Release number of selected component (if applicable):
gdm-3.30.2-2.fc30.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Add nomodeset to boot parameter
2. Wait

Actual results:
Booting hangs right when it should show gdm screen.

Expected results:
gdm should work with basic video driver just fine.

Additional info:
The freeze happens while still at plymouth screen, but I am no longer able to switch plymouth to text output, nor switch to a different TTY.

Comment 1 Fedora Blocker Bugs Application 2019-02-26 12:44:07 UTC
Proposed as a Blocker for 30-beta by Fedora user frantisekz using the blocker tracking app because:

 The boot menu for all supported installer and live images should include an entry which causes both installation and the installed system to use a generic, highly compatible video driver (such as 'vesa'). This mechanism should work correctly, launching the installer or desktop and attempting to use the generic driver.

Comment 2 Geoffrey Marr 2019-03-04 18:26:50 UTC
Discussed during the 2019-03-04 blocker review meeting: [1]

The decision to classify this bug as an "AcceptedBlocker" was made as it violates the following criteria:

The boot menu for all supported installer and live images should include an entry which causes both installation and the installed system to use a generic, highly compatible video driver (such as 'vesa'). This mechanism should work correctly, launching the installer or desktop and attempting to use the generic driver.

We would like to have further testing to verify this behavior.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2019-03-04/f30-blocker-review.2019-03-04-17.03.txt

Comment 3 Chris Murphy 2019-03-06 21:52:31 UTC
I wonder if this is related to bug 1628495. nomodeset hasn't worked on my laptop since Fedora 28, it broke at some point during Fedora 29 and in that bug we never figured out why and it was decided not a blocker because a. not enough hardware was affected b. they tended to work OK without nomodeset.

Anyway, I'm not clear what the expectation is: nomodeset should work, or if it should only work if needed as a fallback.

Comment 4 Chris Murphy 2019-03-06 23:01:13 UTC
On my HP Spectre laptop; F29 and F30 fail to startup when nomodeset is set. However, I actually get *less* information in the journal with F30 failure than F29. In both cases there is no Xorg log produced, but at least in F29 there's a hint "Oct 19 11:02:44 flap.local gnome-shell[932]: Failed to create backend: Could not find a primary drm kms device" but there's nothing at all related to gnome-shell in the F30 case, I have no idea why it's not working nothing useful is logged that I can tell.

In both cases, switching to a shell fails unless I boot with `systemd.debug-shell=1` and then I can switch to tty9.

F29=Booting a USB stick imaged with Fedora-Workstation-Live-x86_64-29-1.2.iso
F30=Booting a USB stick imaged with Fedora-Workstation-Live-x86_64-30-20190224.n.0.iso

Comment 5 Ray Strode [halfline] 2019-03-11 14:11:52 UTC
So this isn't a GDM issue really.  GDM just starts the display server, it doesn't have any influence on if the display server can initialize.

the native mutter backend (wayland) won't work with nomodeset, full stop.  it intrinsically depends on kernel modesetting.

xorg fallback, in theory can work in some cases, some of the time.  I don't think it can work in every case, but I'll defer to X team to decide if this is a bug in X.  Please attach journalctl output !

Comment 6 Adam Williamson 2019-03-11 18:18:28 UTC
could there potentially be an issue here that X fallback isn't working? I believe the intention here is that passing 'nomodeset' causes X fallback to kick in automatically, but maybe that's what broke?

Comment 7 Ray Strode [halfline] 2019-03-11 18:20:36 UTC
pretty unlikely, since comment 0 says "It doesn't make a difference if I force gdm to use Xorg."

Comment 8 Ray Strode [halfline] 2019-03-11 18:26:37 UTC
can you boot to runlevel 3?  log into the getty and run startx ?

Comment 9 Kamil Páral 2019-03-12 08:56:40 UTC
FTR, the intention is to have a safety boot option that works when the default graphics driver doesn't (which is unfortunately still somewhat common). It's now implemented through nomodeset (which in the past always caused Xorg to start up instead of Wayland), but it doesn't have to be. If there are options how to e.g. force llvmpipe to be used (which can be used even with Wayland), and it would be more compatible/less problematic than nomodeset, we can of course switch the implementation for that.

Comment 10 Jeremy Linton 2019-03-12 15:41:11 UTC
Part of this sounds suspiciously similar to bug https://bugzilla.redhat.com/show_bug.cgi?id=1679801, which also appeared a while back in F29, where GDM won't start on seat's that aren't tagged CanGraphical.

Comment 11 Adam Jackson 2019-03-12 17:45:02 UTC
This patches fixes it for me:

-----
diff -up systemd-239/src/login/71-seat.rules.in.jx systemd-239/src/login/71-seat.rules.in
--- systemd-239/src/login/71-seat.rules.in.jx	2018-06-22 07:11:49.000000000 -0400
+++ systemd-239/src/login/71-seat.rules.in	2019-03-12 13:43:11.469554522 -0400
@@ -16,6 +16,9 @@ SUBSYSTEM=="graphics", KERNEL=="fb[0-9]*
 SUBSYSTEM=="drm", KERNEL=="card[0-9]*", TAG+="seat", TAG+="master-of-seat"
 SUBSYSTEM=="usb", ATTR{bDeviceClass}=="09", TAG+="seat"
 
+# allow efifb / uvesafb to be a master if KMS is disabled
+SUBSYSTEM=="graphics", KERNEL=="fb[0-9]", PROGRAM="/usr/bin/grep -qw nomodeset /proc/cmdline", TAG+="master-of-seat"
+
 # 'Plugable' USB hub, sound, network, graphics adapter
 SUBSYSTEM=="usb", ATTR{idVendor}=="2230", ATTR{idProduct}=="000[13]", ENV{ID_AUTOSEAT}="1"
 
-----

Comment 12 Zbigniew Jędrzejewski-Szmek 2019-03-12 19:28:45 UTC
Let's continue the discussion in https://github.com/systemd/systemd/pull/11980 until something is merged upstream.

Comment 13 Adam Williamson 2019-03-13 18:39:15 UTC
Well, something was merged upstream...

Comment 14 Adam Williamson 2019-03-14 19:00:55 UTC
Zbigniew, since this is marked as an accepted blocker, can we get the fix backported to f30 and an update submitted? thanks!

Comment 15 Fedora Update System 2019-03-15 06:30:33 UTC
systemd-241-3.gitc1f8ff8.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-82fd459dae

Comment 16 František Zatloukal 2019-03-15 13:56:44 UTC
systemd-241-3.gitc1f8ff8.fc30 doesn't seem to fix the issue.

I've double-checked the "/usr/lib/udev/rules.d/71-seat.rules" file, it does indeed contain:

"SUBSYSTEM=="graphics", KERNEL=="fb[0-9]", IMPORT{cmdline}="nomodeset", TAG+="master-of-seat""

So, is it possible that this is not enough to fix it? I've tried it in VM and on bare metal. The problem persists, unchanged.

Comment 17 Adam Williamson 2019-03-15 15:52:01 UTC
Same result here. I got the live image openQA built with the systemd update, booted it in a VM in 'basic graphics' mode, it fails just as before.

I am wondering if the 'improved' udev rule is actually valid / correct at all. The docs - https://linux.die.net/man/7/udev - are not particularly clear on what it *means* to stick an IMPORT in the middle of a 'matching' rule like this, but it's not a pattern I can see used anywhere else in any other udev rule and I'm not actually sure it's valid.

The most common pattern for using IMPORT is something like:

IMPORT{cmdline}="nomodeset"
ENV{nomodeset}=="1", TAG+="master-of-seat"

or something along those lines. You also see it as the final item after some matching conditions, like this:

SUBSYSTEMS=="pci", IMPORT{builtin}="hwdb --subsystem=pci"

but IMPORT{cmdline} never seems to be used that way, and I don't see *any* other use of IMPORT in this weird way where it seems to be trying to be a matching condition. I'm just not sure it's correct at all.

Perhaps we could try this instead?

IMPORT{cmdline}="nomodeset"
SUBSYSTEM=="graphics", KERNEL=="fb[0-9]", ENV{nomodeset}=="1", TAG+="master-of-seat"

Comment 18 Adam Williamson 2019-03-15 16:27:46 UTC
Hum, that doesn't seem to work either.

Comment 19 Adam Williamson 2019-03-15 16:37:36 UTC
ajax's original version doesn't seem to work either (doing all this testing in a virtio/spice VM ATM, will confirm on bare metal in a minute).

Comment 20 Adam Williamson 2019-03-15 18:16:49 UTC
Seems the same on bare metal. Possibly of note: there *is* no /dev/fb0 (or /dev/fbanythingelse) when I boot with nomodeset, to runlevel 3 anyway. I can run 'startx' and some kind of X session starts up, doesn't use native, modesetting or fbdev drivers, does use the 'vesa' driver, and manages to display a cursor at least.

Also...I tested UEFI, on bare metal. It also doesn't seem to work - I wind up at a flashing cursor (but can switch to a VT and log in to a tty). In this case, there *is* a /dev/fb0 , and udevadm info shows it with the seat and master-of-seat tags, but...I'm still not getting to a desktop. There may be some kind of different issue there - the X log shows an error, apparently from fbdev driver, then immediately dies (apparently not trying vesa driver):

"Cannot run in framebuffer mode. Please specify busIDs          for all framebuffer devices"

HOWEVER, in a UEFI VM, the fix does seem to work - I get to GDM (autologin doesn't work).

So to summarize:

BIOS VM    - no /dev/fb0, not fixed
BIOS metal - no /dev/fb0, not fixed
UEFI VM    - /dev/fb0, FIXED
UEFI metal - /dev/fb0, not fixed (possibly a different issue, though?)

Comment 21 Adam Williamson 2019-03-15 18:18:07 UTC
In the 'UEFI VM' case, it seems the fbdev driver works successfully.

Comment 22 Adam Williamson 2019-03-15 18:26:47 UTC
Fedora kernel config appears to have CONFIG_FB_VESA=y but CONFIG_FB_UVESA not set. I'm not sure what the implications of that are.

Comment 23 Zbigniew Jędrzejewski-Szmek 2019-03-16 16:25:17 UTC
> I am wondering if the 'improved' udev rule is actually valid / correct at all. The docs - https://linux.die.net/man/7/udev - are not particularly clear on what it *means* to stick an IMPORT in the middle of a 'matching' rule like this, but it's not a pattern I can see used anywhere else in any other udev rule and I'm not actually sure it's valid.

Yeah, the docs certainly could use an improvement. I looked at udevd sources to verify that IMPORT{cmdline}=... works as a conditional too.

> BIOS VM    - no /dev/fb0, not fixed
> BIOS metal - no /dev/fb0, not fixed
> UEFI VM    - /dev/fb0, FIXED
> UEFI metal - /dev/fb0, not fixed (possibly a different issue, though?)

For the first two cases, this patch has no effect.
For the fourth case, could you run 'udevadm info /dev/fb0'? If it has 'master-of-seat' in TAGS, then the issue is something else.

> Fedora kernel config appears to have CONFIG_FB_VESA=y but CONFIG_FB_UVESA not set. I'm not sure what the implications of that are.

The docs for FB_UVESA say:
          This is the frame buffer driver for generic VBE 2.0 compliant
          graphic cards. It can also take advantage of VBE 3.0 features,
          such as refresh rate adjustment.

          This driver generally provides more features than vesafb but
          requires a userspace helper application called 'v86d'. See
          <file:Documentation/fb/uvesafb.txt> for more information.

          If unsure, say N.

It doesn't sound like we need this, but who knows.

Comment 24 Adam Williamson 2019-03-16 17:20:45 UTC
"For the fourth case, could you run 'udevadm info /dev/fb0'? If it has 'master-of-seat' in TAGS, then the issue is something else."

I already did that:

"In this case, there *is* a /dev/fb0 , and udevadm info shows it with the seat and master-of-seat tags, but...I'm still not getting to a desktop."

"It doesn't sound like we need this, but who knows."

I was looking at because of the comment in the original patch from ajax:

"# allow efifb / uvesafb to be a master if KMS is disabled"

note 'uvesafb' there.

Comment 25 Fedora Update System 2019-03-16 20:17:56 UTC
systemd-241-3.gitc1f8ff8.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-82fd459dae

Comment 26 Zbigniew Jędrzejewski-Szmek 2019-03-18 13:08:18 UTC
https://koji.fedoraproject.org/koji/taskinfo?taskID=33608156 is a kernel build with CONFIG_FB_UVESA=m, in case anyone else wants to test that.

Comment 27 Adam Williamson 2019-03-18 15:57:39 UTC
That's no use, because we don't have the helper (/sbin/v86d) - nothing in Fedora seems to include it. Without it, the module doesn't load.

Comment 28 Adam Williamson 2019-03-18 18:20:07 UTC
So, we are now only a few days from the first Beta go/no-go, so this is putting the Beta release at risk. Zbigniew, Adam, will it be possible to come up with a fix in the next day or two?

Comment 29 Zbigniew Jędrzejewski-Szmek 2019-03-19 11:42:38 UTC
I'm sorry, but my knowledge about graphics drivers is nil.

Currently, I don't think this is a systemd bug (after the master-of-seat patch has been merged). The only change in systemd that I'm aware of was not to set the tag, and now we've effectively reverted it in this case.

I'm getting even worse results than adamw:
> UEFI VM    - /dev/fb0, FIXED

I see /dev/fb0 with :seat:master-of-seat:, but gnome-shell on X fails to start with "(EE) Screen(s) found, but none have a usable configuration.". I get the same result with "glx" and "VGA". I *do* get a display with "virtio" graphics.

Hmmmmm, when I *remove* xorg-x11-drv-qxl and xorg-x11-drv-vesa, to force xorg-x11-drv-fbdev to be used, I get a screen (both "qxl" and "VGA" cases).
So it seems that the hand-off from qxl to fbdev X driver is not working. I'll reassign to get more info.

Maybe "vesa" and "qxl" drivers should blacklist themselves if "nomodeset" is present on the kernel command line?

Comment 30 Zbigniew Jędrzejewski-Szmek 2019-03-19 11:44:50 UTC
Created attachment 1545576 [details]
Xorg log with QXL virtual hardware (bad)

Comment 31 Zbigniew Jędrzejewski-Szmek 2019-03-19 11:45:46 UTC
Created attachment 1545578 [details]
Xorg log with QXL virtual hardware with vesa driver uninstalled (bad)

Comment 32 Zbigniew Jędrzejewski-Szmek 2019-03-19 11:46:09 UTC
Created attachment 1545579 [details]
Xorg log with QXL virtual hardware with vesa and qxl drivers uninstalled (good)

Comment 33 Zbigniew Jędrzejewski-Szmek 2019-03-19 11:46:37 UTC
Created attachment 1545580 [details]
Xorg log with virtio virtual hardware (good)

Comment 34 Zbigniew Jędrzejewski-Szmek 2019-03-19 11:46:55 UTC
Created attachment 1545581 [details]
Xorg log with VGA virtual hardware (bad)

Comment 35 Zbigniew Jędrzejewski-Szmek 2019-03-19 11:47:28 UTC
Created attachment 1545582 [details]
Xorg log with VGA virtual hardware with qxl driver uninstalled (bad)

Comment 36 Zbigniew Jędrzejewski-Szmek 2019-03-19 11:47:50 UTC
Created attachment 1545583 [details]
Xorg log with VGA virtual hardware with qxl and vesa drivers uninstalled (good)

Comment 37 Adam Williamson 2019-03-19 17:39:59 UTC
vesa is supposed to work with nomodeset, I think.

Comment 38 Adam Williamson 2019-03-21 17:35:08 UTC
For the record, everyone who tested on F29 reported failure of 'basic graphics' to work (IIRC), but F29 cannot have had this bug at least at release, because it had a systemd build from July 2018 and the systemd commit that stops tagging framebuffer devices as master-of-seat was only merged in October 2018. I verified that an F29 original release live image marks /dev/fb0 as master-of-seat...but nomodeset still doesn't work, in a BIOS VM. On that F29 live image, it *does* work in a UEFI VM...

Comment 39 Adam Williamson 2019-03-21 17:35:49 UTC
I guess the fact that there's no /dev/fb0 at all in a BIOS 'nomodeset' boot could still be the problem in f29? When did the GNOME stack actually start needing something to be 'master-of-seat' before it would start up at all?

Comment 40 Adam Williamson 2019-03-21 17:41:51 UTC
OK, answering my own question here: it in fact only does that in Fedora, I think, because it's done by this patch:

https://src.fedoraproject.org/rpms/gdm/blob/master/f/0001-local-display-factory-defer-initialization-for-CanGr.patch

which has never been merged upstream, it exists as a merge request:

https://gitlab.gnome.org/GNOME/gdm/merge_requests/37

but there's been extensive discussion of it and it has not yet been merged.

Per https://gitlab.gnome.org/GNOME/gdm/merge_requests/37#note_436062 , Ubuntu instead uses a hacky 'solution' which waits either until a master-of-seat device appears, or for ten seconds, then proceeds. So they would not have this problem.

Comment 41 Adam Williamson 2019-03-21 17:48:44 UTC
Ray, could we modify the gdm patch to do something like Canonical's hack - if it waits ten seconds and no seat shows up as CanGraphical, just go ahead anyway and try?

Comment 42 Ray Strode [halfline] 2019-03-21 18:14:23 UTC
i'm tempted to drop the patch entirely for now.

it's not upstream yet, and clearly causing problems.

I don't even remember why i put it in early before it got upstreamed, and my commit message is pretty terse:

commit f517b81933bfa55c904da9c02daef61e99247109
Author: Ray Strode <rstrode@redhat.com>
Date:   Sat Oct 6 09:46:25 2018 -0400
    - Fix login screen for machines that boot to fast

I didn't even spell too right...

Comment 43 Adam Williamson 2019-03-21 18:59:42 UTC
Well, at least before doing that I'd be curious to look into Ubuntu's workaround and see why they introduced that. Perhaps they did actually run into a real-world case where this was a problem?

Comment 44 Adam Williamson 2019-03-21 20:02:54 UTC
FWIW I can't get BIOS nomodeset to work in a VM even with the gdm patch reverted. With virtio graphics, it fails in X with the fb-based drivers failing to find a device (obviously) and vesa driver saying it's ignoring a device with a bound kernel module. qxl graphics fails in gnome-shell with "Failed to create backend: No GPUs found with udev". 'vga' graphics fails the same way as qxl.

Just tested bare metal: it also hit the "No GPUs found with udev" error.

Comment 45 Adam Williamson 2019-03-21 20:05:05 UTC
The "No GPUs found with udev" error appears to come from mutter, specifically, this commit:

https://gitlab.gnome.org/GNOME/mutter/commit/1def09904763fa36d3b388924ee5c9cb6702ba8c

It looks like that can't deal with how things look with 'nomodeset' either, so that probably needs fixing too before 'nomodeset' on BIOS would work.

Comment 46 Adam Williamson 2019-03-21 20:08:57 UTC
At the Fedora 30 Beta go/no-go meeting today:

https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2019-03-21/f30-beta-go_no_go-meeting.2019-03-21-17.00.html

it was agreed that 'basic graphics mode' has lowered in importance significantly that it should no longer block Beta releases. We will decide later whether to bump the requirement to Final or remove it entirely, but as it was agreed that the requirement will no longer be in effect at Beta, this bug is now rejected as a Beta blocker.

For now I'll put it on the proposed Final blocker list, and also propose it as a Beta FE in case of further slips.

Comment 47 Kamil Páral 2019-03-21 20:14:25 UTC
(In reply to Adam Williamson from comment #44)
> FWIW I can't get BIOS nomodeset to work in a VM even with the gdm patch
> reverted.

Some time ago I believe it was Ajax who told me that nomodeset is not supposed to be used in VMs, but only on bare metal. So I suggest only testing this on bare metal machines, otherwise we might see a lot of confusing behavior which is not relevant for the end user anyway.

Comment 48 Adam Williamson 2019-03-22 01:35:52 UTC
It did work with a VM and F28, I believe.

Comment 49 Adam Williamson 2019-03-22 19:35:07 UTC
So halfline and I dug a bit further into the failures I found even with the CanGraphical stuff reverted, and we know what's going wrong there too. I've filed https://bugzilla.redhat.com/show_bug.cgi?id=1691909 for that problem.


Note You need to log in before you can comment on or make changes to this bug.