Bug 1069828 - systemd-journald Failed to write entry X bytes ignoring Bad address
Summary: systemd-journald Failed to write entry X bytes ignoring Bad address
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 22
Hardware: x86_64
OS: Linux
unspecified
low
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-02-25 17:57 UTC by Mikhail Strizhov
Modified: 2016-07-19 20:05 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-19 20:05:12 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Mikhail Strizhov 2014-02-25 17:57:18 UTC
Description of problem:

Lots of error messages in the output of dmesg: 

[   58.191577] systemd-journald[323]: Failed to write entry (24 items, 583 bytes), ignoring: Bad address
[   58.191833] systemd-journald[323]: Failed to write entry (20 items, 551 bytes), ignoring: Bad address
[   58.192162] systemd-journald[323]: Failed to write entry (22 items, 581 bytes), ignoring: Bad address
[   58.192411] systemd-journald[323]: Failed to write entry (23 items, 574 bytes), ignoring: Bad address
[   58.208390] systemd-journald[323]: Failed to write entry (22 items, 591 bytes), ignoring: Bad address
[   58.208643] systemd-journald[323]: Failed to write entry (22 items, 594 bytes), ignoring: Bad address
[   58.237825] systemd-journald[323]: Failed to write entry (26 items, 667 bytes), ignoring: Bad address
[   58.238154] systemd-journald[323]: Failed to write entry (23 items, 581 bytes), ignoring: Bad address


I tried to restart systemd-journald.service

# systemctl restart systemd-journald.service

But the errors still persist:

dmesg output: 

[  277.685602] systemd-journald[323]: Received SIGTERM
[  277.686095] systemd-journald[323]: Failed to write entry (17 items, 459 bytes), ignoring: Bad address
[  277.803599] systemd-journald[1018]: Vacuuming done, freed 0 bytes
[  277.803617] systemd-journald[1018]: Vacuuming done, freed 0 bytes
[  277.803994] systemd-journald[1018]: Can't write entry: Bad address
[  277.804319] systemd-journald[1018]: Failed to write entry (16 items, 456 bytes), ignoring: Bad address
[  277.822822] systemd-journald[1018]: Failed to write entry (17 items, 460 bytes), ignoring: Bad address

Output of /var/log/messages:

Feb 25 10:48:52 dolores systemd: Stopping Journal Service...
Feb 25 10:48:52 dolores systemd-journald[323]: Failed to write entry (22 items, 598 bytes), ignoring: Bad address
Feb 25 10:48:52 dolores systemd-journald[323]: Received SIGTERM
Feb 25 10:48:52 dolores systemd-journald[323]: Failed to write entry (17 items, 459 bytes), ignoring: Bad address
Feb 25 10:48:52 dolores systemd: Starting Trigger Flushing of Journal to Persistent Storage...
Feb 25 10:48:52 dolores systemd: Started Trigger Flushing of Journal to Persistent Storage.


Version-Release number of selected component (if applicable):

systemd-208-14.fc20.x86_64
systemd-devel-208-14.fc20.x86_64
systemd-libs-208-14.fc20.i686
systemd-libs-208-14.fc20.x86_64

Linux dolores 3.13.3-201.fc20.x86_64 #1 SMP Fri Feb 14 19:08:32 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux





How reproducible:

check out the output of dmesg or /var/log/messages

Steps to Reproduce:
1. 
2.
3.

Actual results:

Lots of errors in dmesg.


Expected results:

No errors.


Additional info:

Comment 1 Zbigniew Jędrzejewski-Szmek 2014-03-12 02:27:32 UTC
Hm, can you provide the complete boot log (journalctl -b)? Also, how much free space do you have on the filesystem containing /var/log/? What does 'journalctl --verify' say?

Comment 2 Reilly Hall 2014-05-06 15:38:47 UTC
I am getting the same errors and the journal will no longer show what the syslog and dmesg show.

When I run journalctl --verify this is what I get:

% journalctl --verify                           
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1000
df7f68: invalid object entry array item (4624/6318): 3c3e6e6f69746163░░░░░░░   4%
Invalid object contents at df7f68: Bad message
File corruption detected at /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system:df7f68 (of 134217728 bytes, 10%).
FAIL: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system (Bad message)
53bbe8: invalid object entry array item (1152/2106): a1836f4ae32ab67a░░░░░░░   0%
Invalid object contents at 53bbe8: Bad message
File corruption detected at /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system~:53bbe8 (of 8388608 bytes, 65%).
FAIL: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system~ (Bad message)
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system~
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1000
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1000
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-989
fe9530: invalid object entry array item (855/2106): 676e656c28206669░░░░░░░░  11%
Invalid object contents at fe9530: Bad message
File corruption detected at /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system~:fe9530 (of 58720256 bytes, 28%).
FAIL: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system~ (Bad message)
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1001
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1001
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1000
61ee78: invalid object entry array item (1582/6318): d904000515080106░░░░░░░   0%
Invalid object contents at 61ee78: Bad message
File corruption detected at /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system:61ee78 (of 134217728 bytes, 4%).
FAIL: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system (Bad message)
11f6d58: invalid object entry array item (4178/6318): 401010000000d░░░░░░░░░   6%
Invalid object contents at 11f6d58: Bad message
File corruption detected at /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system:11f6d58 (of 134217728 bytes, 14%).
FAIL: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system (Bad message)
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system~
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1000.journal        
443240: invalid object entry array item (23/26): 77207361206e6f6d░░░░░░░░░░░   0%
Invalid object contents at 443240: Bad message
File corruption detected at /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1000:443240 (of 8388608 bytes, 53%).
FAIL: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1000 (Bad message)
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1001.journal        
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1000
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-989
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1001
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1000
345ecc0: invalid object entry array item (4710/6318): e51ab0000002e░░░░░░░░░  35%
Invalid object contents at 345ecc0: Bad message
File corruption detected at /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system:345ecc0 (of 78233600 bytes, 70%).
FAIL: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/system (Bad message)
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1000
PASS: /var/log/journal/0f3d62f0338a4544b94e29934697a30a/user-1000

Comment 3 Sergio Basto 2014-10-21 14:31:38 UTC
I got same problem after clone my hard disk , I removed  
rm -rf /var/log/journal/0f3d62f0338a4544b94e29934697a30a
and restart the service 
systemctl restart systemd-journalctl

Comment 4 Stefan Becker 2014-12-31 14:18:34 UTC
Looks like systemd-journald really doesn't like disk cloning while it is still running :-( Something to remember for the next upgrade.

BTW: you don't have to delete all journal data to fix this. Just force a log rotation with SIGUSR2, i.e.

  $ kill -SIGUSR2 <pid of systemd-journald>

and the error disappears. See also "man systemd-journald.service".

Comment 5 Fedora End Of Life 2015-05-29 11:03:29 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 6 Fedora End Of Life 2015-06-29 19:05:53 UTC
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 7 John Gotts 2015-12-02 22:45:23 UTC
The problem still exists in Fedora 22.

journalctl --verify

flagged several dozen files with this same error, bad message.

I don't see how

kill -SIGUSR2 <pid of systemd-journald>

is doing anything. Once I removed all of the corrupted files and ran this command again I still have dozens of files in /var/log/journal/.../ dating back to September and October. I also have these gems:

-rw-r-----+ 1 root systemd-journal   8388608 Jun  3 14:38 user-1003.journal
-rw-r-----+ 1 root systemd-journal   8388608 Apr 29  2015 user-1002.journal
-rwxr-xr-x+ 1 root systemd-journal   8388608 Apr  6  2015 user-42.journal
-rwxr-xr-x+ 1 root systemd-journal   8388608 Jan  1  2015 user-991.journal

Are these files dated months ago supposed to be there?

This bug is a bad one. I plugged in a USB device and wanted to see whether it connected as USB 2.0 or USB 3.0. Nope, dmesg just had:

[ 5478.800203] systemd-journald[439]: Failed to write entry (21 items, 650 bytes), ignoring: Bad address

dmesg has a positively gigantic buffer, but this message was being logged every 500 microseconds. I wish I was joking. No kernel messages.

I think the real bug is that systemd needs to stay out of whatever produces dmesg these days. dmesg since the 1990s has been the primary way to get kernel messages, when you do things like plug in USB devices. dmesg is not the place for systemd internal errors. If there is a bad file in the aforementioned subdirectory, ignore it, don't generate a log entry every 500 microseconds.

Hopefully we can get this problem ironed out.

Comment 8 Fedora End Of Life 2016-07-19 20:05:12 UTC
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.