Bug 1017375 - systemd-journald exited with status=1/FAILURE under heavy log load - machine locks up shortly afterwards
Summary: systemd-journald exited with status=1/FAILURE under heavy log load - machine ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 19
Hardware: i686
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-10-09 18:07 UTC by Orion Poplawski
Modified: 2013-11-13 02:25 UTC (History)
7 users (show)

Fixed In Version: systemd-201-2.fc18.9
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-11-13 02:25:09 UTC
Type: Bug


Attachments (Terms of Use)

Description Orion Poplawski 2013-10-09 18:07:20 UTC
Description of problem:

Got this in /var/log/messages:
Oct  9 11:27:22 inferno systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE
Oct  9 11:27:22 inferno systemd[1]: Started Trigger Flushing of Journal to Persistent Storage.

About 5 minutes later the machine completely locked up, had to power cycle.

This system is a firewall and logs a steady stream of packet blocks.  Looks like it got hit by some kind of DOS type attack just now:


-rw-r-----+ 1 root systemd-journal 25M Oct  3 09:06 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  4 11:26 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  4 13:09 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  5 17:22 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  7 04:58 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  8 02:59 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  9 07:31 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  9 10:20 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  9 10:37 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  9 10:52 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  9 11:04 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  9 11:14 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  9 11:24 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  9 11:27 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 25M Oct  9 11:30 journal/19dd186975c4e0486b6469e0e4ce500c/system
-rw-r-----+ 1 root systemd-journal 24M Oct  9 12:00 journal/19dd186975c4e0486b6469e0e4ce500c/system.journal


Version-Release number of selected component (if applicable):
systemd-204-16.fc19.i686

How reproducible:
This seems to be the second time this machine has locked up recently, although I don't see any systemd messages from the previous lockup.

Comment 1 Orion Poplawski 2013-10-09 18:12:35 UTC
The journal has some more messages:

Oct 09 11:27:22 inferno.cora.nwra.com systemd-journal[26743]: Allowing system journal files to grow to 387.3M.
Oct 09 11:27:22 inferno.cora.nwra.com systemd-journal[26743]: Journal started
Oct 09 11:27:22 inferno.cora.nwra.com systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE
Oct 09 11:27:22 inferno.cora.nwra.com systemd[1]: Started Trigger Flushing of Journal to Persistent Storage.
Oct 09 11:27:23 inferno.cora.nwra.com systemd-journal[26743]: Missed 170 kernel messages
Oct 09 11:27:23 inferno.cora.nwra.com systemd-journald[341]: Got invalid event from epoll.
Oct 09 11:30:01 inferno.cora.nwra.com CROND[26749]: (root) CMD (/usr/lib/sa/sa1 1 1)
Oct 09 11:30:28 inferno.cora.nwra.com systemd-journal[26743]: Missed 6 kernel messages
Oct 09 11:30:28 inferno.cora.nwra.com systemd-journal[26743]: Missed 17 kernel messages

Hmm, did we end up with two systemd-journal processes?

Comment 2 Lennart Poettering 2013-10-13 19:35:14 UTC
I am pretty sure this is the issue fixed with this:

http://cgit.freedesktop.org/systemd/systemd/commit/?id=5843c5ebb4341382ae9c87e93c2c87467e573548

Comment 3 Fedora Update System 2013-10-22 16:21:02 UTC
systemd-204-17.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/systemd-204-17.fc19

Comment 4 Fedora Update System 2013-10-23 03:32:52 UTC
Package systemd-204-17.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-204-17.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-19730/systemd-204-17.fc19
then log in and leave karma (feedback).

Comment 5 Fedora Update System 2013-10-25 01:52:53 UTC
systemd-204-17.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 6 Fedora Update System 2013-10-29 16:05:46 UTC
systemd-201-2.fc18.9 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/systemd-201-2.fc18.9

Comment 7 Fedora Update System 2013-10-30 01:53:30 UTC
Package systemd-201-2.fc18.9:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-201-2.fc18.9'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-20267/systemd-201-2.fc18.9
then log in and leave karma (feedback).

Comment 8 Fedora Update System 2013-11-13 02:25:09 UTC
systemd-201-2.fc18.9 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.