RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1550575 - sd-journal implementation opens too many journal files, leading to journal consumers lacking file descriptors
Summary: sd-journal implementation opens too many journal files, leading to journal co...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: systemd
Version: 7.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: systemd-maint
QA Contact: qe-baseos-daemons
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-01 13:48 UTC by Renaud Métrich
Modified: 2021-02-15 07:35 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-15 07:35:35 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Reproducer: dummy program opening the journal and waiting forever (492 bytes, text/x-csrc)
2018-03-01 13:48 UTC, Renaud Métrich
no flags Details

Description Renaud Métrich 2018-03-01 13:48:32 UTC
Created attachment 1402536 [details]
Reproducer: dummy program opening the journal and waiting forever

Description of problem:

This is a continuation of BZ 1540132 (rsyslogd not logging to /var/log/*).
It appears that the sd-journal implementation is very basic, causing 1 file descriptor to be consume for every file in the journal directory, even if these files will never be used later.
When there are many files, e.g.
- because of many users and with mode SplitMode!=none
- because of many rotated files, which can happen if system is alive since a long time

this leads to sd-journal consumers (e.g. rsyslogd/imjournal) to lack from file descriptors.

This is an issue since sd-journal consumers have to be "tuned" one by one and even with tuning, we can never know whether this will work in the future (on long living systems).


Version-Release number of selected component (if applicable):

systemd-219 (RHEL7), systemd-233 (Fedora26)


How reproducible:

Always


Steps to Reproduce:
1. Create persistent journal

  # mkdir -p /var/log/journal
  # systemctl restart systemd-journald

2. Create 2000 users

  # for i in $(seq 1 2000); do useradd user$i; passwd --stdin user$i <<< pass; done

3. Create the journal files for the users (e.g. connect to the system as the user using ssh)

4. Run reproducer in attachment


Actual results:

All filedescriptors from reproducer (default limit is 1024) are consumed by sd-journal

Comment 2 Lukáš Nykrýn 2018-03-01 15:50:40 UTC
Hmm, to be honest, I am not sure what to do here.

Comment 3 Renaud Métrich 2018-03-01 15:52:09 UTC
Maybe change default mode from SplitMode=uid to SplitMode=none would already be an improvment.

Then, open the journal file "upon need" (if it's possible, I really don't know).

Comment 4 Lukáš Nykrýn 2018-03-05 10:30:09 UTC
Could you also open a new issue on upstream github? Maybe we should talk about it there.

Comment 7 RHEL Program Management 2021-02-15 07:35:35 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.


Note You need to log in before you can comment on or make changes to this bug.