Bug 1235598

Summary:	journald logs on overcloud nodes should be persistent
Product:	Red Hat OpenStack	Reporter:	Derek Higgins <derekh>
Component:	rhosp-director	Assignee:	Derek Higgins <derekh>
Status:	CLOSED ERRATA	QA Contact:	Omri Hochman <ohochman>
Severity:	medium	Docs Contact:
Priority:	high
Version:	Director	CC:	jslagle, markmc, mburns, ohochman, rhel-osp-director-maint, rrosa, tdunnon, ykawada
Target Milestone:	ga	Keywords:	Triaged
Target Release:	Director
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	instack-undercloud-2.1.2-15.el7ost	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-08-05 13:56:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Derek Higgins 2015-06-25 09:54:34 UTC

On rhel 7, by default, Journal stores log files only in memory or a small ring-buffer in the /run/log/journal/ directory. 

Which for the default install will end up being 5% of the memory of the server but more importantly they arn't persisted following a reboot.

I think we should be enabling persistent logs in our default images so they are kept following a reboot. 

This should be just a matter of 
mkdir -p /var/log/journal

Comment 3 Derek Higgins 2015-06-25 10:42:10 UTC

I'm raising the priority on this because to fix this post GA for an already deployed environments would require one of
  o Redeployment of new images
  o Document how to make the journal logs persistent across a deployment,
    for each node:
        mkdir -p /var/log/journal
        systemctl restart systemd-journald

Since to fix this now shouldn't be a large chunk of work I think it makes sense to try and get it in

Comment 4 Mark McLoughlin 2015-06-26 06:22:52 UTC

See also https://access.redhat.com/solutions/696893 "How to enable persistent logging for the systemd journal"

In https://access.redhat.com/articles/1346583 there's a reference to persistent logging being the default for RHEL Atomic Host. See also bug #1189872

Comment 7 Omri Hochman 2015-07-06 19:59:27 UTC

failed_qa : instack-undercloud-2.1.2-11.el7ost.noarch   

couldn't find the folder /var/log/journal - doesn't exist.

Comment 8 James Slagle 2015-07-06 21:34:26 UTC

(In reply to Omri Hochman from comment #7)
> failed_qa : instack-undercloud-2.1.2-11.el7ost.noarch   
> 
> couldn't find the folder /var/log/journal - doesn't exist.

to confirm, this directory didn't exist on the overcloud nodes?

can you link to the images you used?

Comment 9 James Slagle 2015-07-07 00:42:41 UTC

ok i see the problem is now that the script to create this dir was missing a +x. will fix in a new build.

Comment 11 Omri Hochman 2015-07-10 20:27:05 UTC

Verified with : instack-undercloud-2.1.2-17.el7ost.noarch
----------------------------------------------------------

files exist on the overcloud nodes :
[root@overcloud-compute-0 ~]# less /var/log/journal/e6f98fb919f344c6b51d108e1f6995c2/
system.journal     user-1000.journal

Comment 13 errata-xmlrpc 2015-08-05 13:56:00 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549