Bug 1171719 - RFE: journal: automatically rotate the file if it is unlinked
Summary: RFE: journal: automatically rotate the file if it is unlinked
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 20
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Jan Synacek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-12-08 13:22 UTC by xset1980
Modified: 2015-03-17 01:53 UTC (History)
10 users (show)

Fixed In Version: systemd-216-21.fc21
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-02-15 03:05:30 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description xset1980 2014-12-08 13:22:15 UTC
Description of problem:

Problem:

journald log all to a system.journal file if is deleted or is corrupted, systemctl status systemd-journald show me "active, running" in green colour like "is all ok" but journalctl say me "no log" and no recover itself so, all logs stops, like: secure, auditd, messages, etc. The workaround is in journald.conf set fordward to syslog, so rsyslog still logging and regenerate a new system.journal file that say "a log was deleted" when journalctl is consulted. From my point of view is a design error that journald stops all OS logs, included secure, auditd log, just because the system.journal file was corrupted or deleted and need rsyslog as backup. Journald uses 7mb of RAM on 32 bit, and rsyslog 3 total 10mb for the single work of logging is my crazy head or is a design error?.

In addition, systemctl status systemd-journald lie me about the status of journald, so, is no reliable for query the daemon.

If the motivation for "no log" is a security reason like "no log == was hacked", makes no sense, because using rsyslog with journald, when the system.journal file is regenerated, say me that the log was initiated in X date, so is obviously that I was hacked.

I propose that journald add a feature like, recover itself in case of deletion of fail of the file, and yes, notify about the event to the root or a previously configured Email.


Version-Release number of selected component (if applicable):

Fedora 20 x86 XFCE, systemd 208-28, system up to date december 8 2014

How reproducible:

Always


Steps to Reproduce:

Just install Fedora 20 x86 spin, delete system.journal file, and stay, the log was never regenerated, just a reboot regenerate it, so, rsyslog is needed... two daemons for the same thing. systemctl status systemd-journald after file deletion.


Actual results:

Journald stops all logs in the system after a lof file deletion.

Expected results:

Journald still logging or autorecover like rsyslog does

Additional info:

Comment 1 Zbigniew Jędrzejewski-Szmek 2014-12-08 13:52:20 UTC
If you remove the output file, or overwrite parts of the file, the daemon cannot really do anything about that: if an attacker has permissions to modify the output file, she has already won. The daemon could check some of those things, but that would be rather inefficient, and not really useful for anything. If you want, you can set up a periodic cryptographic signature with 'journalctl --setup-keys'. See the journalctl man page.

Comment 2 xset1980 2014-12-08 16:06:14 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #1)
> If you remove the output file, or overwrite parts of the file, the daemon
> cannot really do anything about that: if an attacker has permissions to
> modify the output file, she has already won. The daemon could check some of
> those things, but that would be rather inefficient, and not really useful
> for anything. If you want, you can set up a periodic cryptographic signature
> with 'journalctl --setup-keys'. See the journalctl man page.

But rsyslog does. In some cases, If I was hacked for a script kiddie that only put a perlbot or something easy to removal, since the intrusion until my access all logs are losted because journald does not still logging. If rsyslog does, why journald does not?. Is no possible modify some code and make the still logging and maybe alert about corruption?. 

Hack is not the only case, disk failure is a scenario too.

Comment 3 xset1980 2014-12-08 16:13:13 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #1)
> If you remove the output file, or overwrite parts of the file, the daemon
> cannot really do anything about that: if an attacker has permissions to
> modify the output file, she has already won. The daemon could check some of
> those things, but that would be rather inefficient, and not really useful
> for anything. If you want, you can set up a periodic cryptographic signature
> with 'journalctl --setup-keys'. See the journalctl man page.

In addition, systemctl show me active, running, so, something wrong is happening with the journald design, because systemctl check that is running but no if doing the work of journal actively.

If an attacker can delete or modify rsyslog, it still working. OpenSUSE has by default journald+rsyslog fordward enabled, so, is shortcoming of journald, that cannot recover itself after a log deletion like rsyslog does.

Comment 4 Nix\ 2014-12-08 16:22:21 UTC
Well, I have the same question. Why systemd-journald can not recover himself after a write failure or deletion in the hypothetical scene that xset1980 present, while the old rsyslog can do it?. If rsyslog works and recover the log error scene of systemd-journald, it shows that systemd-journald have a void. I think that is a good idea fix it for a future to remove definitively syslog when all software was migrated to systemd design.

Regards

Comment 5 Zbigniew Jędrzejewski-Szmek 2014-12-08 17:10:18 UTC
Can you be more specific in what is not working for you?

Comment 6 xset1980 2014-12-08 21:58:46 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #5)
> Can you be more specific in what is not working for you?

That systemd-journald can't still logging after a log corruption or deletion, and stops *all* logs:

secure.log, auditd.log, messages

While rsyslog never had this problem. And IS necessary for the good function of systemd-journald make fordward to rsyslog (that is not the default in fedora config), because systemd-journald only still the logging to system.journal if rsyslog is receiving the data from systemd-journald.

So, is a bug that systemd-journald needs rsyslog for recover in failure case of log corruption or deletion. Rsyslog is self-sufficient, systemd-journald is not.

So, I have to do the modification to the default journald.config for fordward to rsyslog and prevent an journald stop logging in failure case, what is dirty.

In addition, some control is not working, because systemctl status say "active running" and is not, is not logging anymore after log corruption or deletion, and I stay 1 hour and nothing, journald no auto regenerate himself

Comment 7 Zbigniew Jędrzejewski-Szmek 2014-12-08 22:01:10 UTC
Please, don't go into multi-paragraph interpretation of what you think is happening. Just say what you did, and what happened, and what you think should have happened instead.

Comment 8 Lennart Poettering 2014-12-09 01:50:52 UTC
Well, if you delete log or journal files while they are being written, then of course, they will stay deleted, there's nothing much one can do about it. I mean, if you break logging, you break logging, what did you expect? 

Also journald is not involved in auditd.log, that completely bypasses it.

Comment 9 xset1980 2014-12-09 10:33:43 UTC
(In reply to Lennart Poettering from comment #8)
> Well, if you delete log or journal files while they are being written, then
> of course, they will stay deleted, there's nothing much one can do about it.
> I mean, if you break logging, you break logging, what did you expect? 
> 
> Also journald is not involved in auditd.log, that completely bypasses it.

Lennart Poetteting,

I can understand perfectly. But, systemd-journald does not still logging while systemctl status show me that is working. If i use rsyslog for fordward, still logging after deletion, so, is a bug of systemd-journald that can't still logging after a log deletion, working alone without rsyslog that I want replace.

Comment 10 xset1980 2014-12-09 10:34:51 UTC
Poettering, sorry, typo error.

Comment 11 xset1980 2014-12-09 10:36:35 UTC
Maybe for a language barrier you can't understand me. systemd-journald + rsyslog, still logging after log deletion, systemd-journald not, and stops all log activity, in addition, systemctl status systemd-journald show me that is working.

Comment 12 Zbigniew Jędrzejewski-Szmek 2014-12-09 14:28:02 UTC
So, once again, what steps exactly did you take, and what happened, and what did you expect?

Comment 13 xset1980 2014-12-09 19:06:45 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #12)
> So, once again, what steps exactly did you take, and what happened, and what
> did you expect?

Steps for reproduce the bug:

1.- Start Fedora 20
2.- log as root (su - root)
3.- cd /path/to/the/system.journal file
4.- rm -f system.journal
5.- systemctl status systemd-journald (show that is running and working)
6.- ls -ls (no logs)
7.- journalctl -l (no logs)
8.- Again, systemctl status systemd-journald (show that is active and running)
9.- journalctl -l (no logs)
10.- tailf /var/log/secure and stay
11.- Open a terminal, su - root and log as root
12.- The event is no registered in /var/log/secure or /var/log/messages

Result:

After a log deletion, accidentally or not, systemd-journald stops logging, no regenerate a system.journal file never, stops the log to secure.log and messages additionally.

Wanted for a logging daemon like systemd-journald:

After a log deletion, accidentally or not, that systemd-journald still logging systemd.journal or at least secure.log and /var/log/messages without need fordward to rsyslog daemon, that is the only way in that systemd-journald still logging, recover himself and notify about the log deletion.

Duplicate daemons/effort for the same thing are required today to obtain systemd-journald stability and reliability to errors, and against failures.


Proposal of solution:

systemd-journald must be capable to recover himself against a failure without the need to use rsyslog, a future deprecated daemon with lees features than systemd-journald

Clearly a bug:

systemctl status systemd-journald output shows "active, running", but is not logging nothing, so, is a failed diagnostic of status.

Kind Regards

Comment 14 Nix\ 2014-12-09 19:15:14 UTC
Same erratic and failure behavior here. Fedora 19, 20, 21 beta and CentOS 7 Bare Metal.

Comment 15 Jóhann B. Guðmundsson 2014-12-09 22:03:37 UTC
Remind me again how rsyslog/syslog-ng gracefully recover themselves when the log process continues to log to an unlinked file but still open ( usually due to failed log rotation )?

Comment 16 Jóhann B. Guðmundsson 2014-12-09 22:59:14 UTC
As you can see the same recovering methods work just fine with systemd's journal as they do when this happens with rsyslog or syslog-ng 

Notice how the test to the journal appears as to be expected "uncorrupted" and in the journal output. 

Lennart,Zbyszek perhaps it's best to document this recovery procedure in journalctl man page encase for whatever reason the .journal files remain unlinked but still open. 

If so I can file a patch for it.

# rm /var/log/journal/c77b6b4203be4c8c98e0a09b588f50b5/system.journal

# lsof | grep system.journal
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
systemd-j 3509          root  DEL       REG                8,4              1706048 /var/log/journal/c77b6b4203be4c8c98e0a09b588f50b5/system.journal
systemd-j 3509          root   11u      REG                8,4   8388608    1706048 /var/log/journal/c77b6b4203be4c8c98e0a09b588f50b5/system.journal (deleted)

# systemd-cat -t "TEST" /bin/echo "Systemd journal crash test"

# kill -19 3509

# cat /proc/3509/fd/11 > /var/log/journal/c77b6b4203be4c8c98e0a09b588f50b5/recovered.journal

#kill -9 3509

# ls /var/log/journal/c77b6b4203be4c8c98e0a09b588f50b5/system.journal 
/var/log/journal/c77b6b4203be4c8c98e0a09b588f50b5/system.journal

# journalctl -n 12
-- Logs begin at Thu 2013-10-24 11:47:22 GMT, end at Tue 2014-12-09 22:39:51 GMT. --
Dec 09 22:37:31 localhost.localdomain TEST[3551]: Systemd journal crash test
Dec 09 22:38:21 localhost.localdomain systemd-journal[3558]: Permanent journal is using 2.8G (max allowed 4.0G, trying to leave 4.0G free of 31.4G available → current limit 4.0
Dec 09 22:38:21 localhost.localdomain systemd[1]: systemd-journald.service holdoff time over, scheduling restart.
Dec 09 22:38:21 localhost.localdomain systemd[1]: Stopping Journal Service...
Dec 09 22:38:21 localhost.localdomain systemd[1]: Starting Journal Service...
Dec 09 22:38:21 localhost.localdomain systemd[1]: Started Journal Service.
Dec 09 22:38:21 localhost.localdomain systemd-journal[3558]: Journal started
Dec 09 22:38:21 localhost.localdomain systemd[1]: systemd-journald.service: main process exited, code=killed, status=9/KILL
Dec 09 22:38:21 localhost.localdomain systemd[1]: Unit systemd-journald.service entered failed state.
Dec 09 22:38:21 localhost.localdomain systemd[1]: Starting Trigger Flushing of Journal to Persistent Storage...
Dec 09 22:38:21 localhost.localdomain systemd[1]: Started Trigger Flushing of Journal to Persistent Storage.
Dec 09 22:39:51 localhost.localdomain PackageKit[3251]: daemon quit

Comment 17 Zbigniew Jędrzejewski-Szmek 2014-12-09 23:56:31 UTC
Actually I think there might be a good idea here... If journald would treat the removal of a journal file it is writing to as a signal to rotate the file, some things would become more robust. And rm /var/log/journal/* would simply nuke all logs without interrupting anything. Should be easy to implement with inotify.

Comment 18 Jóhann B. Guðmundsson 2014-12-10 00:19:41 UTC
Well you will need to cat all files that got deleted and are still being written to an new recovery journal files ( or the systemd.journal file etc ) so you dont lose any logs right?

I was thinking this could be better handled by using open() [1] and linkat() [2] which has been there since what 3.11 and try to relink the bloody thing but I assumed you guys had already looked into that possibility and dismissed it as viable solution. 

If we can solve this gracefully well then it puts another plus with journal over alternative log solution since last time I checked they had not fixed this problem ;) 

1. http://man7.org/linux/man-pages/man2/open.2.html
2. http://man7.org/linux/man-pages/man2/linkat.2.html

Comment 19 Jóhann B. Guðmundsson 2014-12-10 00:52:33 UTC
Anyway until such solution get's implemented here's 3 step guide to "survive" for the reporters ;)

Step one 

Figure out which pid is holding the file descriptor (fd)

# lsof | grep system.journal

Step two 

Dump the current state of the file and continue reading the data and dump it into the new file as it's being written into the file

# cat < /proc/1234/fd/5 | /var/log/journal/<foo>/recovered.journal

Step three 

Kill the running process

# kill -9 1234

Comment 20 Jóhann B. Guðmundsson 2014-12-10 01:02:15 UTC
Sorry ignore step two I'm more used to pipe to netcat then to another file

Simply run  cat < /proc/1234/fd/5 > recover.journal && kill -9 1234 or whatever floats your boat

Comment 21 Zbigniew Jędrzejewski-Szmek 2014-12-10 02:25:11 UTC
(In reply to Jóhann B. Guðmundsson from comment #18)
> Well you will need to cat all files that got deleted and are still being
> written to an new recovery journal files ( or the systemd.journal file etc )
> so you dont lose any logs right?
No, I don't think we should try to do that. Just close the old file (it might unlinked, or it might be moved, we don't really care and cannot check), and open a new one.

Comment 22 Jóhann B. Guðmundsson 2014-12-10 10:03:33 UTC
Despite what those reporters claim here this problem is hardly new nor limited to the journal for us administrators in the field ( which is why Lennart closed this as WONTFIX )  and I think it would be better to just leave things as they are and have administrator do what I have proposed manually so they can recover their log *unless* the loss of log lines is within acceptable range ( or none et all ) as well as the implemented solution does not cause increased resources usage.

Comment 23 xset1980 2014-12-10 16:42:56 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #17)
> Actually I think there might be a good idea here... If journald would treat
> the removal of a journal file it is writing to as a signal to rotate the
> file, some things would become more robust. And rm /var/log/journal/* would
> simply nuke all logs without interrupting anything. Should be easy to
> implement with inotify.

you got it. Is the fault / bug that I exposed here, journald is not all it should be a robust system logging for a Linux distro or a server in the case of CentOS 7 or RHEL7.

Comment 24 xset1980 2014-12-10 16:47:49 UTC
(In reply to Jóhann B. Guðmundsson from comment #22)
> Despite what those reporters claim here this problem is hardly new nor
> limited to the journal for us administrators in the field ( which is why
> Lennart closed this as WONTFIX )  and I think it would be better to just
> leave things as they are and have administrator do what I have proposed
> manually so they can recover their log *unless* the loss of log lines is
> within acceptable range ( or none et all ) as well as the implemented
> solution does not cause increased resources usage.

Yes, you may want recover the losted log, or not, anyway, the old rsyslog don't recover the log, but recover itself and still logging. Journald actually after rm -f system.journal stops writing and no create a new system.journal file, in addition, /var/log/messages and /var/log/secure stop receiving any data, just auditd still running and writing to the disk.

Maybe a solution is checking state of FD, detect if the file exist or not, if not, then, restart journald service automatically (is the only way for make that journal recover the log capacity).

Comment 25 Zbigniew Jędrzejewski-Szmek 2014-12-10 16:56:49 UTC
(In reply to xset1980 from comment #23)
> you got it. Is the fault / bug that I exposed here, journald is not all it
> should be a robust system logging for a Linux distro or a server in the case
> of CentOS 7 or RHEL7.
Please turn the volume down a bit. I never said anything like that, so far no bugs have been exposed, and this is not slashdot subthread.

(In reply to xset1980 from comment #24)
> (In reply to Jóhann B. Guðmundsson from comment #22)
> > Despite what those reporters claim here this problem is hardly new nor
> > limited to the journal for us administrators in the field ( which is why
> > Lennart closed this as WONTFIX )  and I think it would be better to just
> > leave things as they are and have administrator do what I have proposed
> > manually so they can recover their log *unless* the loss of log lines is
> > within acceptable range ( or none et all ) as well as the implemented
> > solution does not cause increased resources usage.
> 
> Yes, you may want recover the losted log, or not, anyway, the old rsyslog
> don't recover the log, but recover itself and still logging. Journald
> actually after rm -f system.journal stops writing and no create a new
> system.journal file,
Please re-read what Jóhann wrote... Almost everything you wrote in #c24 is wrong.

Comment 26 xset1980 2014-12-10 19:50:53 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #25)
> (In reply to xset1980 from comment #23)
> > you got it. Is the fault / bug that I exposed here, journald is not all it
> > should be a robust system logging for a Linux distro or a server in the case
> > of CentOS 7 or RHEL7.
> Please turn the volume down a bit. I never said anything like that, so far
> no bugs have been exposed, and this is not slashdot subthread.
> 
> (In reply to xset1980 from comment #24)
> > (In reply to Jóhann B. Guðmundsson from comment #22)
> > > Despite what those reporters claim here this problem is hardly new nor
> > > limited to the journal for us administrators in the field ( which is why
> > > Lennart closed this as WONTFIX )  and I think it would be better to just
> > > leave things as they are and have administrator do what I have proposed
> > > manually so they can recover their log *unless* the loss of log lines is
> > > within acceptable range ( or none et all ) as well as the implemented
> > > solution does not cause increased resources usage.
> > 
> > Yes, you may want recover the losted log, or not, anyway, the old rsyslog
> > don't recover the log, but recover itself and still logging. Journald
> > actually after rm -f system.journal stops writing and no create a new
> > system.journal file,
> Please re-read what Jóhann wrote... Almost everything you wrote in #c24 is
> wrong.

I know that it is not slashdot and I'm not with volume high, only I'm happy that you can explain better than me the concept that I want explain about that is a bug, because is a fail that journald stops logging after a file deletion of the log, while other logging daemons does not.

"Actually I think there might be a good idea here... If journald would treat the removal of a journal file it is writing to as a signal to rotate the file, some things would become more robust. And rm /var/log/journal/* would simply nuke all logs without interrupting anything. Should be easy to implement with inotify."

Is a great idea for me.

I understand perfectly the concept of Jóhann wrote. I'm not a developer, and I do not know how rsyslog does for still logging to disk against a log deletion. So, that journald cannot still logging after the same scene, is a regression in the design of the logging daemon.

Journald should still logging after a file log deletion like others more archaic log daemons. If not, is a bug or design flaw. Anyway like systemd-journal-remote was added as feature (that rsyslog has) I think that the tolerance to failures, for log deletion of disk failure, should be a feature in a robust journal daemon, IMHO.

Kind Regards

Comment 27 Zbigniew Jędrzejewski-Szmek 2014-12-10 19:56:17 UTC
(In reply to xset1980 from comment #26)
> because is a fail that journald stops logging after a file
> deletion of the log, while other logging daemons does not.
No, it does not stop logging. Journald and rsyslog behave exactly the same in this regard: they still have the file open and continue to write to it.

> I understand perfectly the concept of Jóhann wrote. I'm not a developer, and
> I do not know how rsyslog does for still logging to disk against a log
> deletion. So, that journald cannot still logging after the same scene, is a
> regression in the design of the logging daemon.
The point is that this is not a regression. rsyslog does exactly the same, afaik.

Comment 28 xset1980 2014-12-11 13:55:06 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #27)
> (In reply to xset1980 from comment #26)
> > because is a fail that journald stops logging after a file
> > deletion of the log, while other logging daemons does not.
> No, it does not stop logging. Journald and rsyslog behave exactly the same
> in this regard: they still have the file open and continue to write to it.

Not the same behavior. Rsyslog stops logging to the deleted file, like rm -f /var/log/messages but after that, still logging to the file /var/log/secure, the access via SSH, TTY, PTS/X, while journald stops all log activity actually in Fedora 20 and 21.

> > I understand perfectly the concept of Jóhann wrote. I'm not a developer, and
> > I do not know how rsyslog does for still logging to disk against a log
> > deletion. So, that journald cannot still logging after the same scene, is a
> > regression in the design of the logging daemon.
> The point is that this is not a regression. rsyslog does exactly the same,
> afaik.

Technically is no a regression because is not over the same piece of software, but is a regression speaking about journald as the definitive replacement for rsyslog

Kind Regards

Comment 29 Zbigniew Jędrzejewski-Szmek 2014-12-11 14:09:32 UTC
(In reply to xset1980 from comment #28)
> (In reply to Zbigniew Jędrzejewski-Szmek from comment #27)
> > (In reply to xset1980 from comment #26)
> > > because is a fail that journald stops logging after a file
> > > deletion of the log, while other logging daemons does not.
> > No, it does not stop logging. Journald and rsyslog behave exactly the same
> > in this regard: they still have the file open and continue to write to it.
> 
> Not the same behavior. Rsyslog stops logging to the deleted file, like rm -f
> /var/log/messages but after that, still logging to the file /var/log/secure,
> the access via SSH, TTY, PTS/X, while journald stops all log activity
> actually in Fedora 20 and 21.
It continues to log the the deleted file. That's how log rotation is implemented... And the same is true for /var/log/secure: if you delete it, you cannot see it, but rsyslog is still logging to it.

> > > I understand perfectly the concept of Jóhann wrote. I'm not a developer, and
> > > I do not know how rsyslog does for still logging to disk against a log
> > > deletion. So, that journald cannot still logging after the same scene, is a
> > > regression in the design of the logging daemon.
> > The point is that this is not a regression. rsyslog does exactly the same,
> > afaik.
> 
> Technically is no a regression because is not over the same piece of
> software, but is a regression speaking about journald as the definitive
> replacement for rsyslog
There's no regression, the behaviour is equivalent.

Comment 30 Jóhann B. Guðmundsson 2014-12-11 14:09:32 UTC
This is not a regression and this issue most commonly happen when logrotation fails for large log files ( like for example access-log or error logs for httpd etc ) when using rsyslog or syslog-ng. 

If you would configure rsyslog to log everything to a single file ( which you can via rsyslogd.conf ) and then decide to delete that file you would have the exact same end result.

Comment 31 xset1980 2014-12-13 04:54:58 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #29)
> (In reply to xset1980 from comment #28)
> > (In reply to Zbigniew Jędrzejewski-Szmek from comment #27)
> > > (In reply to xset1980 from comment #26)
> > > > because is a fail that journald stops logging after a file
> > > > deletion of the log, while other logging daemons does not.
> > > No, it does not stop logging. Journald and rsyslog behave exactly the same
> > > in this regard: they still have the file open and continue to write to it.
> > 
> > Not the same behavior. Rsyslog stops logging to the deleted file, like rm -f
> > /var/log/messages but after that, still logging to the file /var/log/secure,
> > the access via SSH, TTY, PTS/X, while journald stops all log activity
> > actually in Fedora 20 and 21.

> It continues to log the the deleted file. That's how log rotation is
> implemented... And the same is true for /var/log/secure: if you delete it,
> you cannot see it, but rsyslog is still logging to it.

Actually if you delete /var/log/messages file in a CentOS 6.6 system (does not uses systemd and still has support), the file dissapears and does not come back until rsyslog is restarted, but /var/log/secure still logs access to the system via any allowed source predefined.

1.- rm -f /var/log/messages
2.- "stat /var/log/messages" (no file)
3.- tailf /var/log/secure
4.- Opening a terminal or accessing via ssh, the secure log shows the access while /var/log/messages log does not exist and therefore, cannot show anything, until "service rsyslog restart" is executed.
5.- "stat /var/log/messages" (no file), but /var/log/secure carry on receiving data from rsyslog if is not restarted, after restart /var/log/messages is recreated


> > > > I understand perfectly the concept of Jóhann wrote. I'm not a developer, and
> > > > I do not know how rsyslog does for still logging to disk against a log
> > > > deletion. So, that journald cannot still logging after the same scene, is a
> > > > regression in the design of the logging daemon.
> > > The point is that this is not a regression. rsyslog does exactly the same,
> > > afaik.
> > 
> > Technically is no a regression because is not over the same piece of
> > software, but is a regression speaking about journald as the definitive
> > replacement for rsyslog
> There's no regression, the behaviour is equivalent.

Journald stops logging activity to the /var/log/secure file after deleting system.journal and user-* files from the /path/to/log/ID/* until this is restarted executing:

"systemctl systemd-journald restart", after that, the log file is recreated and shows that a previous log was deleted, and that is right for security reasons, and the logging to the file /var/log/secure returns

So, the behaviour is not the same exactly, and journald loosses a functionality compared to all previous log system (r/syslog/ng), that is still sending data to /var/log/secure (for example) after the /var/log/messages file is deleted. 

Actually, journald log daemon cannot still sending data to /var/log/secure after system.journal or user file is deleted for some reason.

Comment 32 Nix\ 2014-12-13 05:54:54 UTC
The same behavior here, using Fedora 21 x86 final release xfce spin. Systemd 216-12.fc21,i686.

Comment 33 Zbigniew Jędrzejewski-Szmek 2014-12-13 14:33:39 UTC
(In reply to xset1980 from comment #31)
> Actually if you delete /var/log/messages file in a CentOS 6.6 system (does
> not uses systemd and still has support), the file dissapears and does not
> come back until rsyslog is restarted, but /var/log/secure still logs access
> to the system via any allowed source predefined.
> 
> 1.- rm -f /var/log/messages
> 2.- "stat /var/log/messages" (no file)
> 3.- tailf /var/log/secure
> 4.- Opening a terminal or accessing via ssh, the secure log shows the access
> while /var/log/messages log does not exist and therefore, cannot show
> anything, until "service rsyslog restart" is executed.
> 5.- "stat /var/log/messages" (no file), but /var/log/secure carry on
> receiving data from rsyslog if is not restarted, after restart
> /var/log/messages is recreated

Well, obviously, if you have two files, and delete the fist one, the second one is still around, and vice versa.

> > > > > I understand perfectly the concept of Jóhann wrote. I'm not a developer, and
> > > > > I do not know how rsyslog does for still logging to disk against a log
> > > > > deletion. So, that journald cannot still logging after the same scene, is a
> > > > > regression in the design of the logging daemon.
> > > > The point is that this is not a regression. rsyslog does exactly the same,
> > > > afaik.
> > > 
> > > Technically is no a regression because is not over the same piece of
> > > software, but is a regression speaking about journald as the definitive
> > > replacement for rsyslog
> > There's no regression, the behaviour is equivalent.
> 
> Journald stops logging activity to the /var/log/secure file after deleting
> system.journal and user-* files from the /path/to/log/ID/* until this is
> restarted executing:
No. journald never logs to /var/log/secure.

(You are confused by the fact that rsyslog gets its messages from the journal, and copies them to /var/log/secure. So if you remove the journal file, rsyslog cannot read those messages, so it stops forwarding them anywhere. If you remove its sources of information its gone, nothing unexpected here.)

Comment 34 xset1980 2014-12-15 16:59:06 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #33)
> (In reply to xset1980 from comment #31)
> > Actually if you delete /var/log/messages file in a CentOS 6.6 system (does
> > not uses systemd and still has support), the file dissapears and does not
> > come back until rsyslog is restarted, but /var/log/secure still logs access
> > to the system via any allowed source predefined.
> > 
> > 1.- rm -f /var/log/messages
> > 2.- "stat /var/log/messages" (no file)
> > 3.- tailf /var/log/secure
> > 4.- Opening a terminal or accessing via ssh, the secure log shows the access
> > while /var/log/messages log does not exist and therefore, cannot show
> > anything, until "service rsyslog restart" is executed.
> > 5.- "stat /var/log/messages" (no file), but /var/log/secure carry on
> > receiving data from rsyslog if is not restarted, after restart
> > /var/log/messages is recreated
> 
> Well, obviously, if you have two files, and delete the fist one, the second
> one is still around, and vice versa.

Sure, but I assumed that /var/log/secure was managed by journald daemon

> > > > > > I understand perfectly the concept of Jóhann wrote. I'm not a developer, and
> > > > > > I do not know how rsyslog does for still logging to disk against a log
> > > > > > deletion. So, that journald cannot still logging after the same scene, is a
> > > > > > regression in the design of the logging daemon.
> > > > > The point is that this is not a regression. rsyslog does exactly the same,
> > > > > afaik.
> > > > 
> > > > Technically is no a regression because is not over the same piece of
> > > > software, but is a regression speaking about journald as the definitive
> > > > replacement for rsyslog
> > > There's no regression, the behaviour is equivalent.
> > 
> > Journald stops logging activity to the /var/log/secure file after deleting
> > system.journal and user-* files from the /path/to/log/ID/* until this is
> > restarted executing:
> No. journald never logs to /var/log/secure.
> 
> (You are confused by the fact that rsyslog gets its messages from the
> journal, and copies them to /var/log/secure. So if you remove the journal
> file, rsyslog cannot read those messages, so it stops forwarding them
> anywhere. If you remove its sources of information its gone, nothing
> unexpected here.)

Yes, really, I thought that journald manages all logs of the Linux box, for a future and complete replacement of rsyslog/syslog/ng, which is a good idea, because two daemons for the same work, looks like a transitional phase. In addition journald+rsyslog: more RAM usage than journald alone.

I can understand now why some logs stop working after primary (now I know that) log deletion.

"RFE: journal: automatically rotate the file if it is unlinked" is not a good idea too; replace 100% rsyslog, when the *auto-rotate if it is unlinked* feature be implemented?.

Kind Regards

Comment 35 Zbigniew Jędrzejewski-Szmek 2014-12-15 17:17:17 UTC
(In reply to xset1980 from comment #34)
> "RFE: journal: automatically rotate the file if it is unlinked" is not a
> good idea too; replace 100% rsyslog, when the *auto-rotate if it is
> unlinked* feature be implemented?.
I can't parse this paragraph.

Comment 36 xset1980 2014-12-21 13:30:36 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #35)
> (In reply to xset1980 from comment #34)
> > "RFE: journal: automatically rotate the file if it is unlinked" is not a
> > good idea too; replace 100% rsyslog, when the *auto-rotate if it is
> > unlinked* feature be implemented?.
> I can't parse this paragraph.

I wanted to say, and shift some texts:

"RFE: journal: automatically rotate the file if it is unlinked" => the changed tittle <> Is a good idea I guess; because it's allow replace 100% rsyslog, when the 'automatically rotate the file if it is unlinked' was implemented rsyslog can be replaced 100%.

Sorry for my horrible grammar. I think that the centralized log in a system.journal is a 50% no good idea, because if the log is corrupted, the data that rsyslog read from the central log is null. So, why not, in future versions, that directly journald replace to rsyslog, sending the journal to secure, messages log?. Right, is not binary, but, maybe a central binary system.journal file and a fordward to secure and others log, the work that rsyslog actually does, replaced totally by journald. Just my conception.

King Regards.

Comment 37 Lennart Poettering 2015-01-05 01:59:25 UTC
journald git will now check each time it logs a message whether the file it is about to log to is already deleted. If so, it rotates and creates a new file. This should solve the issue and make journald robust towards journal file deletion during runtime.

Comment 40 Fedora Update System 2015-02-05 09:46:13 UTC
systemd-216-18.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/systemd-216-18.fc21

Comment 41 Fedora Update System 2015-02-05 09:47:11 UTC
systemd-208-30.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/systemd-208-30.fc20

Comment 42 Fedora Update System 2015-02-05 20:53:22 UTC
systemd-216-20.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/systemd-216-20.fc21

Comment 43 Fedora Update System 2015-02-06 04:02:15 UTC
Package systemd-208-30.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-208-30.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-1753/systemd-208-30.fc20
then log in and leave karma (feedback).

Comment 44 Fedora Update System 2015-02-15 03:05:30 UTC
systemd-216-20.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 45 Fedora Update System 2015-02-21 04:25:30 UTC
systemd-208-30.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 46 Fedora Update System 2015-03-13 09:37:03 UTC
systemd-216-21.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/systemd-216-21.fc21

Comment 47 Fedora Update System 2015-03-17 01:53:23 UTC
systemd-216-21.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.