Bug 1327303

Summary: journal-remote: change owner of /var/log/journal/remote and create /var/lib/systemd/journal-upload
Product: Red Hat Enterprise Linux 7 Reporter: Martin Stefany <martin>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Branislav Blaškovič <bblaskov>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.2CC: bblaskov, martin, systemd-maint-list
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemd-219-21.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 00:53:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Martin Stefany 2016-04-14 17:41:58 UTC
Description of problem:
1. systemd-journal-remote does not support 'Seal':
apr 13 00:24:19 <hostname> systemd-journal-remote[2322]: [/etc/systemd/journal-remote.conf:2] Unknown lvalue 'Seal' in section 'Remote'

2. /usr/lib/tmpfiles.d/systemd-remote.conf of systemd-journal-gateway is setting insufficient permissions for /var/log/journal/remote:

z /var/log/journal/remote 2755 root systemd-journal-remote - -
z /run/log/journal/remote 2755 root systemd-journal-remote - -

which reports:
apr 14 18:22:34 <hostname> systemd-journal-remote[2388]: Failed to open output journal /var/log/journal/remote/<hostname>.journal: Permission denied
apr 14 18:22:34 <hostname> systemd-journal-remote[2388]: Failed to get writer for source <hostname>: Permission denied

I think
z /var/log/journal/remote 2775 root systemd-journal-remote - -
z /run/log/journal/remote 2775 root systemd-journal-remote - -
would be appropriate here.

3. /var/lib/systemd/journal-upload is not created at installation of systemd-journal-gateway and then systemd-journal-upload fails to even start as it cannot create parent dir of default save-state location /var/lib/systemd/journal-upload/state ; ownership as systemd-journal-upload:root is required for that dir just as well

4. [questionable] systemd-journal-upload user created at installation is missing systemd-journal supplementary group and cannot read journal out-of-the-box, so either created user should have supp. group set as systemd-journal or systemd-journal-upload.service file should contain SupplementaryGroups=systemd-journal same as systemd-journal-gatewayd.service does

Version-Release number of selected component (if applicable):
systemd-219-19.el7_2.7.x86_64
systemd-libs-219-19.el7_2.7.x86_64
systemd-sysv-219-19.el7_2.7.x86_64
systemd-journal-gateway-219-19.el7_2.7.x86_64

How reproducible:
Always, see above.

Steps to Reproduce:
See above.

Actual results:
systemd-journal-upload/remote don't work out-of-the-box, see below

Expected results:
systemd-journal-upload/remote should work out-of-the-box once certificates are generated, /etc/systemd/journal-{upload,remote}.conf are configured, and once /var/log/journal on source and /var/log/journal/{,remote} on destination host are created. No additional config should be necessary.

Additional info:
One point is partially mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1267552, too.

Comment 2 Martin Stefany 2016-04-14 18:23:14 UTC
Point 4 is actually fixed in F23 in systemd-222-4.fc23 by https://bugzilla.redhat.com/show_bug.cgi?id=1262743.

Comment 3 Martin Stefany 2016-04-14 20:41:23 UTC
5. additional point from testing: systemd-journal-upload.service should be auto-restating, e.g. as sshd.service is, since restarting or stopping systemd-journal-remote on destination host kills all sessions and systemd-journal-upload.service would remain in failed state

Proposal:
[Service]
ExecStart=/usr/lib/systemd/systemd-journal-upload \
          --save-state
User=systemd-journal-upload
SupplementaryGroups=systemd-journal
PrivateTmp=yes
PrivateDevices=yes
WatchdogSec=20min
Restart=on-failure
RestartSec=42s

Comment 4 Lukáš Nykrýn 2016-04-15 05:35:23 UTC
Next time please file a separate bugzillas for every issue. This makes it hard for us to track whenever everything was fixed or not.

Comment 5 Lukáš Nykrýn 2016-04-15 05:47:15 UTC
for 2 and 3 we need
https://github.com/systemd/systemd/commit/dcdd4411407067fa1e464dc26ab85ae598fcad7d

Comment 7 Branislav Blaškovič 2016-04-21 13:22:42 UTC
qa acking

Comment 9 Martin Stefany 2016-04-21 13:40:07 UTC
Sorry for the trouble, I will follow it in the future tickets.

Anyway, anothen one (6.) would be: https://bugzilla.redhat.com/show_bug.cgi?id=1329246 / https://github.com/systemd/systemd/issues/1387
and that upstream mentions also incorrect remote-<should_be_remote_hostname_but_is_local_IP>.journal file creation, so I will open another one for it, as it really happens to me as well.

And I get also constant:
Apr 21 15:13:12 <hostname> systemd-journal-remote[25320]: Failed to set file attributes: Operation not supported

using default XFS, SELinux Enforcing, etc. so maybe that's also something to have a look too. It happens also with manual fix from https://github.com/systemd/systemd/commit/dcdd4411407067fa1e464dc26ab85ae598fcad7d mentioned in comment 6.

Comment 11 Branislav Blaškovič 2016-09-21 11:48:12 UTC
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: bug 1327303
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [   PASS   ] :: Command 'systemd-tmpfiles --create' (Expected 0, got 0)
:: [   PASS   ] :: Command 'ls -dl /var/lib/systemd/journal-upload' (Expected 0, got 0)
:: [   PASS   ] :: Command 'ls -ld /var/log/journal/remote | grep 'systemd-journal-remote systemd-journal-remote'' (Expected 0, got 0)
:: [   PASS   ] :: Command 'systemctl stop systemd-journal-gatewayd.socket' (Expected 0, got 0)
:: [   LOG    ] :: Duration: 0s
:: [   LOG    ] :: Assertions: 4 good, 0 bad
:: [   PASS   ] :: RESULT: bug 1327303

Verified.

Comment 13 errata-xmlrpc 2016-11-04 00:53:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2216.html