Bug 1295501

Summary: [spec] fail to start when exporting to remote server
Product: Red Hat Enterprise Linux 7 Reporter: M. Scherer <mscherer>
Component: systemdAssignee: systemd-maint
Status: CLOSED CURRENTRELEASE QA Contact: Radka Brychtova <rskvaril>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.2CC: freaky, jrybar, jscotka, jsynacek, msekleta, rskvaril, systemd-maint-list, zpytela
Target Milestone: rcKeywords: Patch
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemd-219-50.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-23 15:17:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Proposed spec file patch to create needed files none

Description M. Scherer 2016-01-04 17:36:00 UTC
Description of problem:
Trying to start systemd-journal-upload, it fail with the following error message:

janv. 04 17:23:54 freeipa.example.org systemd-journal-upload[22353]: Cannot create parent directory of state file /var/lib/systemd/journal-upload/state: Permission denied



The configuration file is rather standard:

$cat /etc/systemd/journal-upload.conf
[Upload]
URL=https://syslog.example.org:19532/
TrustedCertificateFile=/etc/ipa/ca.crt
ServerKeyFile=/etc/pki/tls/private/journal-upload.pem
ServerCertificateFile=/etc/pki/tls/certs/journal-upload.pem

Version-Release number of selected component (if applicable):
systemd-journal-gateway-219-19.el7.x86_64


How reproducible:
each time


Steps to Reproduce:
1. setup systemd to export remotely the log
2. try to start it ( service systemd-journal-upload start )
3.

Actual results:
it fail with the error message I gave

Expected results:
it should work.

Additional info:

It seems a directory /var/lib/systemd/journal-upload owned by  systemd-journal-upload is missing. Creating it fix the problem.

Comment 2 M. Scherer 2016-01-04 17:46:48 UTC
as a side note, this is fixed by http://pkgs.fedoraproject.org/cgit/systemd.git/commit/?id=ffc19a0568815d02408a29a44f39ffd4b2a152e2 in Fedora.

Comment 4 Zdenek Pytela 2016-12-02 14:11:59 UTC
As a workaround, run the following commands:

=====
mkdir /var/lib/systemd/journal-upload
chown systemd-journal-upload: /var/lib/systemd/journal-upload
=====

Seems to be fixed in F21+.

Comment 5 Ferry 2017-03-07 11:42:39 UTC
Are you sure it's fixed in Fedora? Tried to get this working reliably in Fedora 24 last week (all updates installed). Ran into the following issues and don't have it fully (reliably) working yet:

1) The state file didn't automatically get created and permissions needed to be set. Seeing the link you provided has fixes from end of 2015 I doubt they're included in Fedora 24 at least as I ran into the same thing.

2) When running as a service (ran fine as root) SElinux blocks systemd-journal-upload from TCP traffic and it crashes because it can't reach the remote thus. Error from systemd-journal-upload is very non-descriptive:

Mar 07 12:19:59 <hostname> systemd[1]: Started Journal Remote Upload Service.
Mar 07 12:19:59 <hostname> systemd[1]: systemd-journal-upload.service: Main process exited, code=exited, status=1/FAILURE
Mar 07 12:19:59 <hostname> systemd[1]: systemd-journal-upload.service: Unit entered failed state.
Mar 07 12:19:59 <hostname> systemd[1]: systemd-journal-upload.service: Failed with result 'exit-code'.

3) The service is very error prone. By default it's not configured to automatically restart and unlike what one (well I at least) would expect for a log upload service, which might be *very* important in many environments, the systemd-journal-upload process just dies if it can't reach the remote end. If it's down for a couple of minutes, doing updates, network issues, whatever, systemd-journal-upload just dies and that's it. Maybe you'll get your logs if it reboots (if they're persistent), maybe if some user pays attention and manually restarts it (hoping they aren't rotated yet), making it a very far from reliable service in the default setup.

Comment 7 Jan Rybar 2017-08-18 11:33:12 UTC
Created attachment 1315172 [details]
Proposed spec file patch to create needed files

Change in spec file allows creation of vital files during installation.
Also the description of the journal-gatewayd package was extended by contents list.

Comment 8 Jan Rybar 2017-12-04 12:05:08 UTC
Hello,

it seems that the directory /var/lib/systemd/journal-upload is created automatically by tmpfiles.d (there exist a configuration file for that already in RHEL), but it needs to run tmpfiles.d daemon again (e.g. by rebooting).
I can imagine a scenario where a user tries to start the journal-upload daemon right after installation of package systemd-gateway, hence the directory is not created yet and the daemon fails.

To fix the problem, we are going to create the directory during installation of the package.

Meanwhile there is similar problem with systemd-journal-remote which demands more delicate solution and there shall be a new bugzilla opened for it.

Comment 9 Lukáš Nykrýn 2018-02-08 13:45:18 UTC
I forget about this one and now there are only blocker bugs accepted for 7.5. SO let's move it to 7.6.

Comment 10 Lukáš Nykrýn 2018-02-23 15:17:28 UTC
Actually, I did not forget about it and the fix is there since December.