Bug 1984980

Summary:	User logins are super slow when system has tons of disks attached
Product:	Red Hat Enterprise Linux 8	Reporter:	Renaud Métrich <rmetrich>
Component:	systemd	Assignee:	Michal Sekletar <msekleta>
Status:	NEW ---	QA Contact:	Frantisek Sumsal <fsumsal>
Severity:	high	Docs Contact:
Priority:	high
Version:	8.4	CC:	apmukher, dtardon, francois.rigault, msekleta, pdwyer, systemd-maint-list
Target Milestone:	rc	Keywords:	Performance
Target Release:	---	Flags:	apmukher: needinfo? (msekleta)
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Renaud Métrich 2021-07-22 15:08:14 UTC

Description of problem:

When a system has tons of disks attached (~7000), user logins are slow due to "systemd --user" instance reading all the disks.

A customer reported 11 seconds to log in:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
$ time ssh system date
Tue Jul  6 06:39:59 GMT 2021

real    0m11.493s
user    0m0.012s
sys     0m0.004s
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Executing strace, we can see that all block devices (/sys/devices/virtual/block/<device>) are opened and stat'ed, which takes a lot of time in the end.

IMO "systemd --user" needs to be optimized to either not retrieve this information or get it from "systemd --system" upon start.


Version-Release number of selected component (if applicable):

systemd-239-41.el8_3.2.x86_64


How reproducible:

Always on customer site having VxVM volumes.
Cannot reproduce locally using either iscsi or loopback devices

Comment 1 François Rigault 2023-06-25 16:39:35 UTC

> Cannot reproduce locally using either iscsi or loopback devices

there is https://github.com/systemd/systemd/issues/9518 in systemd 239 that prevents systemd --user to list every device, which makes it more difficult to reproduce on RHEL 8, but it's easy to reproduce on any other version (eg RHEL 9.2, Fedora Rawhide)