RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1793533 - systemd sends tons of useless PropertiesChanged messages when a mount happens
Summary: systemd sends tons of useless PropertiesChanged messages when a mount happens
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: systemd
Version: 8.1
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: rc
: 8.0
Assignee: Michal Sekletar
QA Contact: Frantisek Sumsal
URL:
Whiteboard:
Depends On:
Blocks: 1793527 1917516 1920502 1946669 1946671
TreeView+ depends on / blocked
 
Reported: 2020-01-21 14:20 UTC by Renaud Métrich
Modified: 2024-03-25 15:38 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1946669 1946671 (view as bug list)
Environment:
Last Closed: 2020-11-04 02:02:06 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github systemd-rhel rhel-8 pull 113 0 None closed (#1793533) Don't emit PropertiesChanged signal needlessly 2021-02-15 16:15:27 UTC
Github systemd systemd pull 17234 0 None closed unit: don't emit PropertiesChanged signal if adding a dependency to a unit is a no-op 2021-02-15 16:15:29 UTC
Red Hat Product Errata RHSA-2020:4553 0 None None None 2020-11-04 02:02:39 UTC

Internal Links: 1884262

Description Renaud Métrich 2020-01-21 14:20:21 UTC
This bug was initially created as a copy of Bug #1793527

I am copying this bug because: 

Also applies to RHEL 8.1

Description of problem:

It appears that a single "mount" operation on a single file system (including a "mount -o remount") makes systemd send 2 PropertiesChanged message per mount point seen on the system (even if the mount point is not touched).
This creates a huge systemd load, and potentially a huge load on DBus.

The root cause behind this is an absence of caching of the /proc/1/mountinfo table.


Version-Release number of selected component (if applicable):

systemd-219 AND systemd-239 (RHEL8)


How reproducible:

Always


Steps to Reproduce:
1. Set up a system with 200 mount points on a "data" volume group

  # vgcreate data /dev/sdb
  # for i in $(seq 1 200); do lvcreate -n lv$i -L 10M data; mkfs.ext4 /dev/data/lv$i; echo "/dev/mapper/data-lv$i /data/lv$i ext4 defaults 0 0" >> /etc/fstab; mkdir -p /data/lv$i; done
  # systemctl daemon-reload
  # mount -a

2. Monitor systemd's "sendmsg" while performing 100 remounts of /data/lv1

  In a terminal:
  # timeout 1m strace -ttTvyy -o /tmp/systemd.strace -s 1024 -e sendmsg -p 1

  In another terminal:
  # for i in $(seq 1 100); do mount -o remount /data/lv1; sleep 0.1; done

3. Count the number of PropertiesChanged messages sent

  # grep -c PropertiesChanged /tmp/systemd.strace

Actual results:

40600


Expected results:

100 (ideally) or 200 (observed with a patch because there is a PropertiesChanged message for "/" seen, holding "/data/lv1")


Additional info:

The load is due to systemd parsing the whole /proc/1/mountinfo table and UDEV database to construct "description" field of corresponding Device units, requiring tons of malloc().
Finally all this gets sent through DBus "blindly" (no one cares about these messages).

A simple optimization (count dropping to 200) is to only send data when "description" field changes.

A better optimization would be to cache the content of /proc/1/mountinfo and only recompute the "description" field based on UDEV database for the modified mount points, but it's harder to do.

Comment 12 Renaud Métrich 2020-07-03 14:07:31 UTC
See also this discussion Upstream: this storm can lead to DOS ...

https://github.com/systemd/systemd/issues/13674

Comment 35 Renaud Métrich 2020-10-02 15:40:13 UTC
See also this new PR: https://github.com/systemd/systemd/pull/17234

Comment 42 errata-xmlrpc 2020-11-04 02:02:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: systemd security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4553

Comment 46 jdee 2022-02-08 20:10:24 UTC
Was this Systemd patch ported to CoreOS 4.8?

Comment 47 Michal Sekletar 2023-01-30 11:10:29 UTC
(In reply to jdee from comment #46)
> Was this Systemd patch ported to CoreOS 4.8?

Fix for this bug was delivered in RHEL-8.4. CoreOS 4.8 is based on RHEL-8.4 EUS hence it should contain this bugfix.


Note You need to log in before you can comment on or make changes to this bug.