Bug 1244691

Summary: systemd journal remote service does not work when multiple virtual machines send logs via journal upload service
Product: Red Hat Enterprise Linux 7 Reporter: angsak <angsakel>
Component: systemdAssignee: David Tardon <dtardon>
Status: CLOSED ERRATA QA Contact: Frantisek Sumsal <fsumsal>
Severity: low Docs Contact:
Priority: unspecified    
Version: 7.3CC: dtardon, fsumsal, gvohra, jsynacek, martin, misc, rhel, systemd-maint, trae32566, tsantiago
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-29 20:32:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description angsak 2015-07-20 09:50:17 UTC
Description of problem:

I use systemd-journal upload service to send logs from multiple virtual machines (senders) to a single virtual machine (receiver) which uses systemd-journal remote service. There are 9 sender virtual machines sending to 1 receiver virtual machine. 

There is a bug in the code of journal-remote service. Instead of writing each virtual machine's (sender) hostname to the beginning of the .journal  under /var/log/journal/remote, it writes the local hostname (i.e. the receiver virtual machine's hostname and not the sender's hostname.
From testing, it seems that each virtual machine (sender) is trying to write to a single .journal file under /var/log/journal/remote which appears to cause a corrupted journal remote service.
I have also noticed that possibly due to this corruption the disk space (of receiver) is quickly filled (available disk space is about 17G and journalctl --disk-usage in each sender node is pretty low ~30MB so this does not justify such a fast filling of 17G disk space)

By monitoring with journalctl -f each sender VM I also noticed that they did not log locally too fast.


Version-Release number of selected component (if applicable):
systemctl --version
systemd 219


How reproducible:
it is reproducible each time you want to send from many virtual machines to a single one via journal-upload and remote services 


Actual results:
remote-receiver_hostname.internalnet.localdomain@*****.journal

Expected results:
remote-sender1_hostname.internalnet.localdomain@*****.journal
remote-sender2_hostname.internalnet.localdomain@*****.journal

Additional info:

Comment 1 Trae Santiago 2017-04-09 11:14:48 UTC
I'm having this issue as well. It appears this is fixed in:
https://github.com/systemd/systemd/pull/2078

This was initially reported by the same person in:
https://github.com/systemd/systemd/issues/1387

I'm guessing Red Hat hasn't pulled this yet, so it'd probably be a good idea to do that. 

@angsak, I believe this issue may have gotten lost in the ether given that it was filed (incorrectly) under 'Bugtool'. Hopefully, someone can move it so we can get the ball rolling :)

Comment 3 Michael S. 2017-04-09 12:20:15 UTC
Ok so I moved to the correct component, and it did set severity to urgent by itself, and a few stuff regarding 'rc'. I am not sure of the right value, so I let them like this for triaging later.

Comment 4 David Tardon 2019-12-13 15:13:44 UTC
*** Bug 1329279 has been marked as a duplicate of this bug. ***

Comment 5 David Tardon 2019-12-13 16:01:46 UTC
PR: https://github.com/systemd-rhel/rhel-7/pull/62

Comment 7 Lukáš Nykrýn 2020-04-27 13:16:57 UTC
fix merged to github master branch -> https://github.com/systemd-rhel/rhel-7/pull/62

Comment 13 errata-xmlrpc 2020-09-29 20:32:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: systemd security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4007