RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1292447 - journald stop logging
Summary: journald stop logging
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: systemd
Version: 7.2
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: rc
: ---
Assignee: systemd-maint
QA Contact: Branislav Blaškovič
URL:
Whiteboard:
: 1276563 (view as bug list)
Depends On:
Blocks: 1203710 1289485 1313485 1331339 1363687
TreeView+ depends on / blocked
 
Reported: 2015-12-17 13:38 UTC by Kevin Cousin
Modified: 2023-10-06 17:30 UTC (History)
38 users (show)

Fixed In Version: systemd-219-21.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1331339 1363687 (view as bug list)
Environment:
Last Closed: 2016-11-04 00:48:53 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1312146 0 unspecified CLOSED sd_journal_sendv() seems to only allow 511 bytes for binary data 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1632211 0 urgent CLOSED sd_journal_get_cursor() failed - when time is changed 2023-10-06 17:55:46 UTC
Red Hat Knowledge Base (Solution) 2117311 0 None None None 2016-04-08 15:04:11 UTC
Red Hat Product Errata RHBA-2016:2216 0 normal SHIPPED_LIVE systemd bug fix and enhancement update 2016-11-03 13:24:51 UTC

Internal Links: 1312146 1632211

Description Kevin Cousin 2015-12-17 13:38:20 UTC 
Description of problem:
We don't have anyting logged in journal, and so on rsyslog.

Version-Release number of selected component (if applicable):
systemd-219-19.el7.x86_64

How reproducible:
Alawys

Steps to Reproduce:
1.journalctl --verify 
2.systemctm restart systemd-journald
3.journalctl -f 

Actual results:
Logging restart but stop working a few minutes later and journalctl --verify show corrupted files

Expected results:
Logging ok

Additional info:
Comment 1 Kevin Cousin 2015-12-17 13:39:29 UTC 
The issue appears after upgrade to 7.2. Here is dmesg output :

[92658.758413] systemd-journald[17718]: Failed to write entry (21 items, 547 bytes), ignoring: Cannot assign requested address
[92658.818970] systemd-journald[17718]: Failed to write entry (21 items, 888 bytes), ignoring: Cannot assign requested address
[92658.834527] systemd-journald[17718]: Failed to write entry (21 items, 594 bytes), ignoring: Cannot assign requested address
[92658.854182] systemd-journald[17718]: Failed to write entry (21 items, 847 bytes), ignoring: Cannot assign requested address
[92658.870043] systemd-journald[17718]: Failed to write entry (21 items, 580 bytes), ignoring: Cannot assign requested address
[92658.875305] systemd-journald[17718]: Failed to write entry (21 items, 895 bytes), ignoring: Cannot assign requested address
Comment 3 Jerome LE TANOU 2015-12-21 14:46:48 UTC 
Hello,

We have the same problem with an up-to-date RHEL 7.2 Server.

This server hosts a proxy server (squid-3.3.8) and we send all access log to syslog (rsyslog-7.4.7).

After a while, we get no more log and instead we have thousands of journald error lines :
---------------------------------------
#dmesg
...
[333311.366625] systemd-journald[810]: Failed to write entry (20 items, 616 bytes), ignoring: Cannot assign requested address
[333311.401662] systemd-journald[810]: Failed to write entry (20 items, 633 bytes), ignoring: Cannot assign requested address
[333311.476621] systemd-journald[810]: Failed to write entry (20 items, 615 bytes), ignoring: Cannot assign requested address
...
---------------------------------------

---------------------------------------
# systemctl status systemd-journald
● systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled)
   Active: active (running) since Thu 2015-12-17 14:06:59 CET; 3 days ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 810 (systemd-journal)
   Status: "Processing requests..."
   CGroup: /system.slice/systemd-journald.service
           └─810 /usr/lib/systemd/systemd-journald
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
---------------------------------------

---------------------------------------
# journalctl --verify
4f7160: invalid object
File corruption detected at /run/log/journal/54ddca845fdd4b4c95e9da7b01a9b28f/system.journal:4f7160 (of 8388608 bytes, 62%).
FAIL: /run/log/journal/54ddca845fdd4b4c95e9da7b01a9b28f/system.journal (Cannot assign requested address)
PASS: /run/log/journal/54ddca845fdd4b4c95e9da7b01a9b28f/system
...
---------------------------------------

If we removed the corrupted file and restart the service, we receive again logs but the problem appears again after a while

Do you have an idea to permanently correct this dysfunction ? 

Thank you.
Comment 4 Christophe GRENIER 2015-12-21 17:15:12 UTC 
Bonjour ;-)

In the meantime, I am using the following workaround on my servers

/etc/rsyslog.conf
#$ModLoad imjournal # provides access to the systemd journal
$ModLoad imklog # reads kernel messages (the same are read from journald)
#$OmitLocalLogging on
#$IMJournalStateFile imjournal.state

/etc/systemd/journald.conf
[Journal]
#Storage=none
Storage=persistent
ForwardToSyslog=yes


If you choose Storage=none, journald will not try to log anything itself. Use Storage=persistent if you still want to see if journald is ok or not.
Comment 5 Eugen O 2015-12-22 10:47:49 UTC 
The previous workaround did not help in my case.
I had to revert systemd packages + rsyslog. Still this is not a proper solution. I find this issue critical.
Comment 6 Jeff Earickson 2016-02-08 16:04:47 UTC 
This bug is a major pain for us and I hope it get fixed soon.  It seems to happen most often on our web servers (which generate a lot of syslog traffic).

I have found no way to get rsyslog restarted and working again without a reboot.  The usual things of "systemctl rsyslog restart" and "systemctl daemon-reload" do not work.  Any way to recover without a reboot?
Comment 7 Michal Sekletar 2016-02-11 16:19:01 UTC 
More detailed steps to reproduce would be very appreciated. I can't reproduce on up2date RHEL-7.2.
Comment 8 Rick Ellis 2016-02-13 20:33:00 UTC 
I recently tried turning compression off in /etc/systemd/journald.conf. After deleting all the journal files and restarting journald the problem went away.
Comment 9 Michal Sekletar 2016-02-17 08:32:44 UTC 
I've backported possible fix for this issue. In case anyone is willing to test, fell free to grab these test rpms.

http://people.redhat.com/~msekleta/systemd-219-19.el7.0.bz1292447.0.src.rpm/
Comment 10 Jeff Earickson 2016-02-17 19:14:55 UTC 
I ran Michal's rpms yesterday for a while on a test box (you have to download and update all of the rpms), and the system ran fine.  But it puts out very little syslogging, so this is not a fair test.

Redhat put out some systemd patches yesterday, probably related to the glibc bug:

systemd.x86_64                                 219-19.el7_2.4
systemd-libs.x86_64                            219-19.el7_2.4
systemd-sysv.x86_64                            219-19.el7_2.4

I would guess that Michal's fixes are NOT in these updates, correct?
Comment 11 Lukáš Nykrýn 2016-02-18 08:55:11 UTC 
> Redhat put out some systemd patches yesterday, probably related to the glibc
> bug:
> 
> systemd.x86_64                                 219-19.el7_2.4
> systemd-libs.x86_64                            219-19.el7_2.4
> systemd-sysv.x86_64                            219-19.el7_2.4
> 
> I would guess that Michal's fixes are NOT in these updates, correct?
There was nothing journal or glibc related in that update.
Comment 12 Christophe GRENIER 2016-02-19 10:02:00 UTC 
One of my server is running Michal's rpms since yesterday. There are a lot of syslogging and the journal is still ok.
Comment 13 Kevin Cousin 2016-03-07 07:59:08 UTC 
I installed Michal's rpm last week and my logs looks fine.
Comment 14 Michal Sekletar 2016-03-07 12:05:52 UTC 
Thanks for providing test results. Based on your input I am granting devel_ack for 7.3.
Comment 16 Eugen O 2016-03-09 07:36:05 UTC 
Michal's patch seem to solve the issue on my side too.
Comment 18 Jeff Earickson 2016-03-11 16:05:31 UTC 
FWIW, I have noticed a correlation between oom-killer and rsyslog problems a couple of hours later.  We run coldfusion on some web servers, which uses java, which eats memory.  Due to memory leaks, oom-killer has to whack java every so often. systemctl will autorestart coldfusion.  After that, rsyslog will fail within a few hours.
Comment 19 Lukáš Nykrýn 2016-03-16 12:32:09 UTC 
pushed to staging
https://github.com/lnykryn/systemd-rhel/commit/f45b66a348f5778bd391ad1b0a0e09bf5789b415?bz=1292447
https://github.com/lnykryn/systemd-rhel/commit/d205f5f85569e2dddca96362ce2db4e2a0b99d00
-> post
Comment 20 Mike McCune 2016-03-28 23:38:32 UTC 
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Comment 22 Michal Sekletar 2016-04-08 15:16:45 UTC 
*** Bug 1276563 has been marked as a duplicate of this bug. ***
Comment 27 Jeff Earickson 2016-04-18 14:08:13 UTC 
When does Redhat plan to release an official patch for this bug?  It is still killing me and I don't want to run the previously available rpms on critical production systems.  I was hoping a real patch would be out by now....
Comment 34 rhbugzilla 2016-04-27 01:34:44 UTC 
There is a new leak in 219-19.el7_2.7. I installed the patched RPMs linked above and the problem did not go away. I don't know how to tell you to reproduce it, it's a tiny vps with 128 MB of ram, and systemd-journald will slowly eat up that ram fully. If I kill the process or use systemctl to restart it, the memory usage goes back down to normal, and then slowly goes back up.

I don't know what you'd need from me to help with this, but if you let me know how, I can do it (or at least find out how).
Comment 37 Sergey Morozik 2016-05-12 02:14:39 UTC 
Had the same issue.
Workaround worked for me:
echo "Compress=no" >> /etc/systemd/journald.conf
systemctl restart systemd-journald; systemctl restart rsyslog.service
Basically, disabled compression.
Comment 45 Michal Sekletar 2016-06-01 12:23:11 UTC 
Because we can't reproduce this problem internally we need more debugging information from a customer. First I'd start with stracing journald. Please start following command,

strace -o journal-strace.log -tt -p $(pidof systemd-journald)

and in new terminal run "logger foobar" 3 times, then Ctrl-C strace and upload journal-strace.log.
Comment 46 Lukáš Nykrýn 2016-06-01 12:39:12 UTC 
From looking to the list of installed rpms, I would guess that there could be some conflict between journal and some 3rd party security software. Is it possible to try the system with those services turned off?
Comment 51 Pavel Khusainov 2016-06-14 15:09:23 UTC 
I've exactly the same problem on one of my nodes: it stops logging and dmesg is full of "systemd-journald[17718]: Failed to write entry (21 items, 895 bytes), ignoring: Cannot assign requested address".

I've tried to turn of the compression, however it doesn't work for me.

I've made strace as Michal Sekletar suggested, plus lsof and more verbose strace, look here: https://gist.github.com/alvelcom/d3b9e8f201db607018b8736f7688b6aa


---------------------------------------
$ journalctl --verify
2e227a8: invalid object
File corruption detected at /run/log/journal/56745016c55f4dae8b17425e32eba949/system.journal:2e227a8 (of 50331648 bytes, 96%).
FAIL: /run/log/journal/56745016c55f4dae8b17425e32eba949/system.journal (Cannot assign requested address)
7fffee8: unused data (entry_offset==0)
PASS: /run/log/journal/56745016c55f4dae8b17425e32eba949/system
PASS: /run/log/journal/56745016c55f4dae
---------------------------------------

I deleted /run/log/journal/56745016c55f4dae8b17425e32eba949/system.journal, restarted systemd-journald, and after this actions journald seems returing to work.
Comment 68 errata-xmlrpc 2016-11-04 00:48:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2216.html
Comment 69 Plumber Bot 2022-01-21 15:38:23 UTC 
Dropping the stale needinfo. If our input is still needed, please set the needinfo again.
Comment 70 Plumber Bot 2022-01-21 15:38:26 UTC 
Dropping the stale needinfo. If our input is still needed, please set the needinfo again.
Note You need to log in before you can comment on or make changes to this bug.