Bug 1828758

Possibly related upstream issue: https://github.com/systemd/systemd/issues/14730

Red Hat Enterprise Linux 7 shipped it's final minor release on September 29th, 2020. 7.9 was the last minor releases scheduled for RHEL 7.
From intial triage it does not appear the remaining Bugzillas meet the inclusion criteria for Maintenance Phase 2 and will now be closed. 

From the RHEL life cycle page:
https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_2_Phase
"During Maintenance Support 2 Phase for Red Hat Enterprise Linux version 7,Red Hat defined Critical and Important impact Security Advisories (RHSAs) and selected (at Red Hat discretion) Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available."

If this BZ was closed in error and meets the above criteria please re-open it flag for 7.9.z, provide suitable business and technical justifications, and follow the process for Accelerated Fixes:
https://source.redhat.com/groups/public/pnt-cxno/pnt_customer_experience_and_operations_wiki/support_delivery_accelerated_fix_release_handbook  

Feature Requests can re-opened and moved to RHEL 8 if the desired functionality is not already present in the product. 

Please reach out to the applicable Product Experience Engineer[0] if you have any questions or concerns.  

[0] https://bugzilla.redhat.com/page.cgi?id=agile_component_mapping.html&product=Red+Hat+Enterprise+Linux+7

*** Bug 1947190 has been marked as a duplicate of this bug. ***

I'm reopening this because this is a critical issue, not only on systems with many users, but also on systems having heavy docker workloads (e.g. 1500 scopes).

It appears that the **list-unit-files** operations ends up having some kind of O(n^2) algorithm which doesn't scale.
I didn't dig completely into the code but observed from backtrace and reproducer that the check on the files is not linear, which is a real problem for scalability.
I see something like below:

~~~
- method_list_unit_files()
  - for all scope units
    - find_symlinks_in_scope(<scopeX>)
      - for all scope units
        - find_symlinks_fd(<scopeY.d>)
~~~


Simple reproducer using scopes:

1. I start 100 scopes and check the number of **fcntl()** operations

  ~~~
  # for i in $(seq 1 100); do systemd-run --scope --unit="runscope$i" /bin/sleep 3600 & sleep 0.01; done
  
  # strace -ttTvyy -o systemd_100scopes.strace -e fcntl -s 256 -p 1 &
  
  # systemctl list-unit-files >/dev/null
  Failed to list unit files: Connection timed out 
  ~~~
  
  I wait for the operation to complete on the systemd side: even though the `systemctl` command exited in timeout, the operation is still ongoing. Then I check the strace:
  
  ~~~
  # grep -c "F_SETFD, FD_CLOEXEC" systemd_100scopes.strace
  242841
  
  # grep -m1 "F_SETFD, FD_CLOEXEC" systemd_100scopes.strace
  10:36:19.522057 fcntl(16</etc/systemd/system>, F_SETFD, FD_CLOEXEC) = 0 <0.000004>
  # tac systemd_100scopes.strace | grep -m1 "F_SETFD, FD_CLOEXEC"
  10:36:46.140838 fcntl(17</run/systemd/system/user-0.slice.d>, F_SETFD, FD_CLOEXEC) = 0 <0.000004>
  ~~~

  Here above, the number of operations is already crazy (242841) and systemd took 27 seconds to perform the operation on my system (under strace, which slows down the process).
  

2. Now I redo the test with 200 more scopes (total 300 scopes)

  ~~~
  # for i in $(seq 101 300); do systemd-run --scope --unit="runscope$i" /bin/sleep 3600 & sleep 0.01; done
  
  # strace -ttTvyy -o systemd_300scopes.strace -e fcntl -s 256 -p 1 &
  
  # systemctl list-unit-files >/dev/null
  Failed to list unit files: Connection timed out 
  ~~~
  
  Wait for the operation to complete on the systemd side, then check the strace:
  
  ~~~
  # grep -c "F_SETFD, FD_CLOEXEC" systemd_300scopes.strace
  690041

  # grep -m1 "F_SETFD, FD_CLOEXEC" systemd_300scopes.strace
  10:41:31.842921 fcntl(16</etc/systemd/system>, F_SETFD, FD_CLOEXEC) = 0 <0.000005>
  # tac systemd_300scopes.strace | grep -m1 "F_SETFD, FD_CLOEXEC"
  10:42:42.399127 fcntl(17</run/systemd/system/user-0.slice.d>, F_SETFD, FD_CLOEXEC) = 0 <0.000004>
  ~~~

  Here above, the number of operations increases by a 3 factor and systemd took more than one minute to perform the operation on my system (3 times the time for 100 scopes).


Without stracing, with 1700 scopes, I see that systemd is not responsive for more than 35 seconds.

Setting the Customer Flag = Yes; per EN-40415

https://github.com/systemd/systemd/pull/18943 should reduce the time ~3-4 times. I uploaded test rpms that include that patch to http://people.redhat.com/dtardon/systemd/bz1828758-list-unit-files/ , in case anyone wants to try that.

fix merged to github master branch -> https://github.com/redhat-plumbers/systemd-rhel7/pull/135

fix merged to github master branch -> https://github.com/redhat-plumbers/systemd-rhel7/pull/137

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (systemd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0066