Bug 1830896 - Very slow udev after upgrade to fedora 32
Summary: Very slow udev after upgrade to fedora 32
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 32
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1833778 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-04 09:03 UTC by The Source
Modified: 2020-09-10 00:23 UTC (History)
16 users (show)

Fixed In Version: systemd-245.7-1.fc32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-30 18:56:09 UTC
Type: Bug


Attachments (Terms of Use)
journalctl -b 0 --no-hostname output (488.79 KB, text/plain)
2020-05-04 09:03 UTC, The Source
no flags Details
Proposed upstream systemd patch to cache efi variables (1.78 KB, application/mbox)
2020-05-11 22:50 UTC, Olivier Samyn
no flags Details
Patch for systemd-245.4 (fedora 32) (1.27 KB, patch)
2020-05-12 19:10 UTC, Oleg Samarin
no flags Details | Diff

Description The Source 2020-05-04 09:03:49 UTC
Created attachment 1684737 [details]
journalctl -b 0 --no-hostname output

Created attachment 1684737 [details]
journalctl -b 0 --no-hostname output

Created attachment 1684737 [details]
journalctl -b 0 --no-hostname output

Description of problem:
After upgrade to Fedora 32 from 31 the boot process became very long. Udev takes several minutes to detect all devices. After I'm finally able to login, I have to wait for several minutes before network and audio become operational.
Other symptoms: worker threads, mainly efi_rts_wq and unbound_event, load cpu a lot, around 20 instances of systemd-udevd run. Restarting dbus.service removes that cpu usage and udev processes (but that causes gtk based applications start slowly under KDE session).

My hardware:
ASUS N56VB laptop
CPU: Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
RAM: 16GB
HDD: Samsung SSD 860 EVO 2TB

I tried loading kernels left from fedora 31 (5.6.8, 5.6.7), but that didn't help.

Update: plugging or removing usb devices causes kworker threads to eat cpu for several seconds and system does not detect the device (or it's disconnect) for that period

Version-Release number of selected component (if applicable):
systemd-245.4-1.fc32.x86_64

How reproducible:
always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 The Source 2020-05-05 15:14:42 UTC
I narrowed this problem down. It seems that when udev is executing IMPORT{cmdline} operation inside the rule, it takes long time. So removing some unneeded rules containing this operation greatly speeds up boot process. For my system I removed 61-gdm.rules, 64-md-raid-assembly, 65-md-incremental. The last two caused slowdown the most. I moved those 3 files to my home folder and remade the initrd using dracut. Now my system boost much faster.
I consider this a temporary workaround until IMPORT{cmdline} is fixed.

Comment 2 The Source 2020-05-11 07:55:44 UTC
Please check this upstream discussion:
https://github.com/systemd/systemd/issues/15739

It's likely that it is already fixed upstream.

Comment 3 Oleg Samarin 2020-05-11 14:09:32 UTC
All these *.rules files are present in F31 as well. But udev works quickly in F31

Comment 4 Eugine 2020-05-11 17:03:49 UTC
Yes, a confirm than with all these systemd rules F31 boot was much quicker.
Now i have the same problem... udev spends abot 4 minutes on lvm-monitor.service

Comment 5 Olivier Samyn 2020-05-11 22:50:42 UTC
Created attachment 1687479 [details]
Proposed upstream systemd patch to cache efi variables

Applying upstream systemd patch on top of the fedora packages solves most of the issue for me. At least the system is more responsive; devices are detected as quick as it was in fedora 31. Not a I a slow HDD, so I don't know if this solves all speed issues.
But at least it improves things.

Upstream link: https://github.com/systemd/systemd/pull/15627

Comment 6 Olivier Samyn 2020-05-11 23:32:46 UTC
If it can be useful for anyone, I've rebuilt the systemd package with the included patch in a copr repository: https://copr.fedorainfracloud.org/coprs/oleastre/systemd/
Use it at your own risk. It works on my machine but that's the only guarantee I can give.

Comment 7 The Source 2020-05-12 06:24:33 UTC
(In reply to Olivier Samyn from comment #6)
> If it can be useful for anyone, I've rebuilt the systemd package with the
> included patch in a copr repository:
> https://copr.fedorainfracloud.org/coprs/oleastre/systemd/
> Use it at your own risk. It works on my machine but that's the only
> guarantee I can give.

It helped me, thank you.

Comment 8 Eugine 2020-05-12 07:56:45 UTC
(In reply to Olivier Samyn from comment #6)
> If it can be useful for anyone, I've rebuilt the systemd package with the
> included patch in a copr repository:
> https://copr.fedorainfracloud.org/coprs/oleastre/systemd/
> Use it at your own risk. It works on my machine but that's the only
> guarantee I can give.

This systemd patch works for me. Thanx.

The problem was with limits of efivarfs?

Comment 9 Olivier Samyn 2020-05-12 14:41:17 UTC
It seems so, or maybe something to do with some hardware that is slow to answer I don't know. 
As the original poster, I also noticed the kernel worker efi_rts_wq at 100% CPU for a long time. Once that one was down to 0%, the system became usable.

There are maybe other regressions regarding boot time, but at least the one related to devices enumeration and udev, seems to be solved with that patch.

Comment 10 Oleg Samarin 2020-05-12 19:10:40 UTC
Created attachment 1687785 [details]
Patch for systemd-245.4 (fedora 32)

I can confirm that applying the patch to systemd-245.4-1 (present in Fedora32) resolves the problem of booting Fedora32

Comment 11 Oleg Samarin 2020-05-12 19:14:59 UTC
*** Bug 1833778 has been marked as a duplicate of this bug. ***

Comment 12 Gabriele Turchi 2020-05-13 14:00:53 UTC
With https://copr.fedorainfracloud.org/coprs/oleastre/systemd/ on my ASUS N56V no changes...

Comment 13 Olivier Samyn 2020-05-13 15:37:53 UTC
Two questions to have a bit more context: 
- are you sure the systemd package has been updated ? or maybe you can try to re-package systemd with the proposed patch to see.
- Can you give us a bit more info about your symptoms ? is your slowness coming from devices not accessible or something else ?

The original reporter has a N56VB laptop and mine is a N56VZ (seems to be related to some bios versions), so it feel strange it does not change anything on yours.

Comment 14 Gabriele Turchi 2020-05-13 17:33:31 UTC
Sorry, my fault. The first time the upgrade of systemd failed partially (dnf was unable to update the i686 version). 

Now the patched version works well even for me. 

Thank you very much.

Comment 15 The Source 2020-05-21 07:34:23 UTC
There are some new commits upstream that claim to fix the issue (at least upstream issue is closed).

Comment 16 Olivier Samyn 2020-06-06 00:21:36 UTC
I had the bad surprise today to get a systemd upgrade to version 245.6. That new version does not include the efivars cache patch, and of course it overrides my own compiled version. So, for the one relying on my copr, please wait before doing an update, I'll port the patch tomorrow.

If there is a systemd package maintainer listening, would it be possible to add the proposed patch (efi variables caching) in the fedora package to at least avoid the impacted systems to be unresponsive for 10 minutes ? or to help me get it in the upstream systemd-stable branch ? 

I'm participating in the upstream issue to test some solutions while they are enhancing other related efi variables issues.
But the proposed initial patch is already merged in systemd master, and it's required for impacted systems.

Let's try to avoid another systemd update without this issue fixed.

Comment 17 Fedora Update System 2020-07-27 09:52:00 UTC
FEDORA-2020-2faf839786 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-2faf839786

Comment 18 Fedora Update System 2020-07-28 15:19:16 UTC
FEDORA-2020-2faf839786 has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-2faf839786`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-2faf839786

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 19 Fedora Update System 2020-07-30 18:56:09 UTC
FEDORA-2020-2faf839786 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.