Bug 1818429

Summary: Fedora 32 Hangs at Boot on udev initialization
Product: [Fedora] Fedora Reporter: Joshua <hagtosmo>
Component: systemdAssignee: Orphan Owner <extras-orphan>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 32CC: aivaras.laimikis, code, extras-orphan, fedoraproject, filbranden, flepied, gmagic, info, jonathan, lnykryn, m.koshelev, msekleta, ssahani, s, systemd-maint, thuryn1, yuwatana, zbyszek, z
Target Milestone: ---Keywords: Bugfix
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-18 18:10:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Picture of issue
none
Kernel logs
none
Boot log with udev log set to debug none

Description Joshua 2020-03-28 02:30:13 UTC
Description of problem:
When booting all boot processes work fine till it reaches udev than it hangs with a message stating that the system is waiting for device initialization. After over 50 seconds the system than continues to boot.

Version-Release number of selected component (if applicable):
N/A

How reproducible:
This happens every time my system boots.

Steps to Reproduce:
1.Push power button on PC
2.Wait for udev to initialize the device after 50 seconds
3.Device boots after 50 seconds normally

Actual results:
Total boot time over 2 minutes including typing in password for encryption.

Expected results:
Boot should be no more than 30 seconds total (based on other distros I have used on this pc.

Additional info:
My Hardware info hope this helps.

H/W path                      Device      Class          Description
====================================================================
                                          system         X470 AORUS GAMING 5 WIF
/0                                        bus            X470 AORUS GAMING 5 WIF
/0/0                                      memory         64KiB BIOS
/0/9                                      memory         16GiB System Memory
/0/9/0                                    memory         [empty]
/0/9/1                                    memory         8GiB DIMM DDR4 Synchron
/0/9/2                                    memory         [empty]
/0/9/3                                    memory         8GiB DIMM DDR4 Synchron
/0/b                                      memory         576KiB L1 cache
/0/c                                      memory         3MiB L2 cache
/0/d                                      memory         16MiB L3 cache
/0/e                                      processor      AMD Ryzen 5 2600 Six-Co
/0/100                                    bridge         Family 17h (Models 00h-
/0/100/0.2                                generic        Family 17h (Models 00h-
/0/100/1.3                                bridge         Family 17h (Models 00h-
/0/100/1.3/0                              bus            Advanced Micro Devices,
/0/100/1.3/0/0                usb1        bus            xHCI Host Controller
/0/100/1.3/0/0/2                          communication  Bluetooth wireless inte
/0/100/1.3/0/0/6              scsi10      storage        USB3.0 Card Reader
/0/100/1.3/0/0/6/0.0.0        /dev/sdf    disk           USB3.0 CRW   -SD
/0/100/1.3/0/0/6/0.0.0/0      /dev/sdf    disk           
/0/100/1.3/0/0/6/0.0.1        /dev/sdg    disk           30GB USB3.0 CRW   -SD
/0/100/1.3/0/0/6/0.0.1/0      /dev/sdg    disk           30GB 
/0/100/1.3/0/0/6/0.0.1/0/2    /dev/sdg2   volume         15EiB Windows FAT volum
/0/100/1.3/0/0/9                          input          USB Laser Game Mouse
/0/100/1.3/0/0/a                          multimedia     Blue Snowball
/0/100/1.3/0/1                usb2        bus            xHCI Host Controller
/0/100/1.3/0.1                scsi0       storage        400 Series Chipset SATA
/0/100/1.3/0.1/0              /dev/sda    disk           256GB SPCC Solid State
/0/100/1.3/0.1/0/1                        volume         599MiB Windows FAT volu
/0/100/1.3/0.1/0/2            /dev/sda2   volume         1GiB EXT4 volume
/0/100/1.3/0.1/0/3            /dev/sda3   volume         236GiB EFI partition
/0/100/1.3/0.1/1              /dev/sdb    disk           500GB Samsung SSD 840
/0/100/1.3/0.1/1/1            /dev/sdb1   volume         465GiB EXT4 volume
/0/100/1.3/0.1/2              /dev/sdc    disk           2TB ST2000DM001-1CH1
/0/100/1.3/0.1/2/1            /dev/sdc1   volume         1863GiB EFI partition
/0/100/1.3/0.1/3              /dev/sdd    disk           2TB ST2000DM001-1CH1
/0/100/1.3/0.1/3/1            /dev/sdd1   volume         1863GiB EXT4 volume
/0/100/1.3/0.1/0.0.0          /dev/sde    disk           2TB Hitachi HUA72302
/0/100/1.3/0.1/0.0.0/1        /dev/sde1   volume         1863GiB EFI partition
/0/100/1.3/0.2                            bridge         400 Series Chipset PCIe
/0/100/1.3/0.2/0                          bridge         400 Series Chipset PCIe
/0/100/1.3/0.2/0/0                        multimedia     Sound Core3D [Sound Bla
/0/100/1.3/0.2/1                          bridge         400 Series Chipset PCIe
/0/100/1.3/0.2/2                          bridge         400 Series Chipset PCIe
/0/100/1.3/0.2/2/0            wlp5s0      network        Wireless-AC 9260
/0/100/1.3/0.2/3                          bridge         400 Series Chipset PCIe
/0/100/1.3/0.2/3/0            enp6s0      network        I211 Gigabit Network Co
/0/100/1.3/0.2/4                          bridge         400 Series Chipset PCIe
/0/100/1.3/0.2/9                          bridge         400 Series Chipset PCIe
/0/100/1.3/0.2/9/0                        bus            ASM1143 USB 3.1 Host Co
/0/100/1.3/0.2/9/0/0          usb3        bus            xHCI Host Controller
/0/100/1.3/0.2/9/0/1          usb4        bus            xHCI Host Controller
/0/100/1.3/0.2/9/0/1/2        scsi9       storage        ASMT1153e
/0/100/1.3/0.2/9/0/1/2/0.0.0  /dev/cdrom  disk           BDDVDRW UH12NS30
/0/100/3.1                                bridge         Family 17h (Models 00h-
/0/100/3.1/0                              display        TU116 [GeForce GTX 1650
/0/100/3.1/0.1                            multimedia     TU116 High Definition A
/0/100/3.1/0.2                            bus            NVIDIA Corporation
/0/100/3.1/0.2/0              usb5        bus            xHCI Host Controller
/0/100/3.1/0.2/1              usb6        bus            xHCI Host Controller
/0/100/3.1/0.3                            bus            TU116 [GeForce GTX 1650
/0/100/7.1                                bridge         Family 17h (Models 00h-
/0/100/7.1/0                              generic        Zeppelin/Raven/Raven2 P
/0/100/7.1/0.2                            generic        Family 17h (Models 00h-
/0/100/7.1/0.3                            bus            Zeppelin USB 3.0 Host c
/0/100/7.1/0.3/0              usb7        bus            xHCI Host Controller
/0/100/7.1/0.3/0/3                        input          Corsair K70R Gaming Key
/0/100/7.1/0.3/1              usb8        bus            xHCI Host Controller
/0/100/8.1                                bridge         Family 17h (Models 00h-
/0/100/8.1/0                              generic        Zeppelin/Renoir PCIe Du
/0/100/8.1/0.2                            storage        FCH SATA Controller [AH
/0/100/8.1/0.3                            multimedia     Family 17h (Models 00h-
/0/100/14                                 bus            FCH SMBus Controller
/0/100/14.3                               bridge         FCH LPC Bridge
/0/101                                    bridge         Family 17h (Models 00h-
/0/102                                    bridge         Family 17h (Models 00h-
/0/103                                    bridge         Family 17h (Models 00h-
/0/104                                    bridge         Family 17h (Models 00h-
/0/105                                    bridge         Family 17h (Models 00h-
/0/106                                    bridge         Family 17h (Models 00h-
/0/107                                    bridge         Family 17h (Models 00h-
/0/108                                    bridge         Family 17h (Models 00h-
/0/109                                    bridge         Family 17h (Models 00h-
/0/10a                                    bridge         Family 17h (Models 00h-
/0/10b                                    bridge         Family 17h (Models 00h-
/0/10c                                    bridge         Family 17h (Models 00h-
/0/10d                                    bridge         Family 17h (Models 00h-
/0/10e                                    bridge         Family 17h (Models 00h-
/0/1                                      system         PnP device PNP0c01
/0/2                                      system         PnP device PNP0b00
/0/3                                      system         PnP device PNP0c02
/0/4                                      system         PnP device PNP0c02
/1                            virbr0      network        Ethernet interface
/2                            virbr0-nic  network        Ethernet interface

Comment 1 Joshua 2020-03-28 02:34:33 UTC
Created attachment 1674218 [details]
Picture of issue

Comment 2 Joshua 2020-04-26 04:53:06 UTC
I am still having this issue and now its worse because after installing nvidia drivers my screen shuts off after booting. After the display gets initialized during boot the monitor says no input detected and goes into power saving mode.

Comment 3 Eugine 2020-05-03 18:48:12 UTC
Can confirm this. After system upgrade to Fedora 32 caught the same problem - long-long-long boot. Boot process hangs on udev Device Initialization. This process seems to last 3 minutes and more (Asus ROG G46VW). Same issue on the fresh updated F32 release + updates from testing repository. There was no such issue on Fedora 31.

There 3 processes that fail cause of long device initialization:
- Failed to start udev Wait for Complete Device Initialization (systemd-udev-settle.service failed which seems to be deprecated)
- long lvm2 monitor (by the way i don't use lvm)
- Failed to start Network Manager Wait Online

Same issue with F32 live-usb.

Even if i mask or disable lvm2-monitor.service or/and systemd-udev-settle.service (this one is called by lvm2-monitor.service), system boots in GDM in 30-50 secs, but with no wi-fi, mouse and keyboard initialized. In 3+ minutes after GDM boot all these devices is initialized.

By the way lvm2-monitor seems to be "Done" but with warnings (WARNING: Device /dev/sdb6 not initialized in udev database even after waiting 10000000 microseconds.) So none of my sdaX or sdbX was initialized in udev database.

Comment 4 Olivier Samyn 2020-05-04 21:09:07 UTC
I confirm the problem, I have 2 laptops, I updated from fedora 31 to 32. On the old one, with an Ivy Bridge processor, it takes more than 5 minutes to initialize any device and load the associated kernel modules.

I tried various settings changes in the bios, or disabling some services, but nothing seems to solve that issue. 

If I wait long enough for all the devices to come up, the system is perfectly usable.

I also tried a live image, and get the same problem.

I'll try to get some kernel logs and attach here to show the problem.

Comment 5 Olivier Samyn 2020-05-04 21:45:37 UTC
Created attachment 1684997 [details]
Kernel logs

As you can see from the attached kernel logs, the system takes almost 10 minutes to load the network and wifi devices.
And my attached USB mouse although early detected, becomes usable after 15 minutes (while the touchpad is available earlier, around the same time as the keyboard)

While trying to debug the issue, I noticed in top, a kernel "kworker:efi_rts_wq" thread running at 100% CPU. And that one disappear after a while (when system is usable, I don't see it any more in ps); don't know if it's related.

I can provide journalctl logs if needed.

Comment 6 Olivier Samyn 2020-05-06 06:32:21 UTC
Created attachment 1685558 [details]
Boot log with udev log set to debug

I did some further testing trying to find a problematic kernel module or something relevant. 
Tried with selinux in permissive mode, not better.
So, here is a boot log with systemd-udevd booted with log_priority=debug.
There are some gaps in device detection, and I have the impression that some devices are detected multiple times.
If someone has tips what to search for, I would be happy to try stuff.

Comment 7 Olivier Samyn 2020-05-11 10:13:50 UTC
This is probably related to: https://bugzilla.redhat.com/show_bug.cgi?id=1830896
I'll make further testing today to verify if the proposed systemd patch works.

Comment 8 Eugine 2020-05-11 17:06:06 UTC
(In reply to Olivier Samyn from comment #7)
> This is probably related to:
> https://bugzilla.redhat.com/show_bug.cgi?id=1830896
> I'll make further testing today to verify if the proposed systemd patch
> works.

will wait for your feedback

(my alternative OS Clear Linux with fresh systemd seems to boot normal)

Comment 9 Zbigniew Jędrzejewski-Szmek 2020-10-08 08:20:31 UTC
This bug was filed against udev package, but that page is logn
gone (and nobody looks at the bugs here). Reassigning
to systemd.

Comment 10 Fedora Program Management 2021-04-29 16:47:44 UTC
This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 11 Zbigniew Jędrzejewski-Szmek 2021-05-18 18:10:06 UTC

*** This bug has been marked as a duplicate of bug 1830896 ***