Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1852845

Summary: [Bug] journald: rate-limiting configuration is not working
Product: Red Hat Enterprise Linux 8 Reporter: Oliver Falk <ofalk>
Component: systemdAssignee: David Tardon <dtardon>
Status: CLOSED DUPLICATE QA Contact: Frantisek Sumsal <fsumsal>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.2CC: dtardon, john.petrucci, msekleta, rhel, systemd-maint-list, yferszt
Target Milestone: rcKeywords: Reopened
Target Release: 8.0Flags: pm-rhel: mirror+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-15 09:22:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1117877    
Bug Blocks:    

Description Oliver Falk 2020-07-01 12:31:17 UTC
Description of problem:
RateLimit configuration in systemd-journald doesn't work.
This configuration was previously working in my customers environment on RHEL7, but now on RHEL 8(.2) doesn't any longer.


Version-Release number of selected component (if applicable): 239-30.el8_2


How reproducible: Always


Steps to Reproduce:
1. add these two lines to /etc/systemd/journald.conf:
    RateLimitBurst=10000
    RateLimitIntervalSec=5s
2. restart journald
    # systemctl restart systemd-journald
3. Create following script and execute it:
    $ cat gen_burst.sh
    #!/bin/bash

    for i in $(seq 1 50000); do
      echo 'test9'
   done
4. Check the number of messages in the journal:
    # journalctl | grep test9 | wc -l
    32500

Actual results:
Too many messages in the journal.


Expected results:
There should be ~ 10k and not this amount.


Additional info:
Attaching customer case. I was able to reproduce this on a rhel8.2 sandbox system.

We found RHBZ#1030529, which looks similar, but I checked the mentioned upstream pr (https://github.com/systemd/systemd/pull/6392) and it seems to be integrated in our systemd sources.

Checking the changelog of the package, I find the following:

    * Thu Oct 24 2019 Lukas Nykryn <lnykryn> - 239-19
    [ ... ]
    - core: implement per unit journal rate limiting (#1719577)
    [ ... ]

This seems to be the latest change to rate limiting - at least from an RPM changelog point of view - but maybe it helps.

Comment 1 David Tardon 2020-07-02 11:49:08 UTC
(In reply to Oliver Falk from comment #0)
> Steps to Reproduce:
> 1. add these two lines to /etc/systemd/journald.conf:
>     RateLimitBurst=10000
>     RateLimitIntervalSec=5s
> 2. restart journald
>     # systemctl restart systemd-journald
> 3. Create following script and execute it:
>     $ cat gen_burst.sh
>     #!/bin/bash
> 
>     for i in $(seq 1 50000); do
>       echo 'test9'
>    done

Huh? This doesn't write anything to the journal. Did you mean "execute it as a service", perhaps?

Anyway, a simple attempt to reproduce this doesn't work:

# systemd-run --wait -p StandardOutput=journal ./gen_burst.sh
# journalctl | grep test9 | wc -l
53

Could we have a more precise reproducer, please?

Comment 2 Oliver Falk 2020-07-02 13:16:20 UTC
Sorry,
 copy-paste issue on my side.

This should have been:

    $ cat gen_burst.sh
    #!/bin/bash

    for i in $(seq 1 50000); do
      echo 'test9'|logger
   done

This should work!

Comment 7 Oliver Falk 2021-01-28 11:18:35 UTC
After mail exchange with Michal:

First, systemd-cat isn't the right way to go - this was a misunderstanding.

Secondly, this needs to be reproduced from systemd unit context (eg. using systemd-run).
However, during testing Michael found out that this actually also doesn't work as expected and therefore we've decided to reopen this BZ.

@msekleta in case I missed anything, please feel free to correct me!

Comment 11 Lukáš Nykrýn 2021-11-23 14:48:21 UTC
We discussed this today at our systemd meeting and it is not easy to move the bug forward. As it is specified in the man page, "This rate limiting is applied per-service, so that two services which log do not interfere with each other's limits."
But in the reproducer, we have no way how to identify that those messages came from the same source.

We would need something like https://bugzilla.redhat.com/show_bug.cgi?id=1117877