Bug 1235598 - journald logs on overcloud nodes should be persistent
Summary: journald logs on overcloud nodes should be persistent
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: Director
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ga
: Director
Assignee: Derek Higgins
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-25 09:54 UTC by Derek Higgins
Modified: 2015-08-05 13:56 UTC (History)
8 users (show)

Fixed In Version: instack-undercloud-2.1.2-15.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-08-05 13:56:00 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Gerrithub.io 237690 None None None Never
Red Hat Product Errata RHEA-2015:1549 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform director Release 2015-08-05 17:49:10 UTC

Description Derek Higgins 2015-06-25 09:54:34 UTC
On rhel 7, by default, Journal stores log files only in memory or a small ring-buffer in the /run/log/journal/ directory. 

Which for the default install will end up being 5% of the memory of the server but more importantly they arn't persisted following a reboot.

I think we should be enabling persistent logs in our default images so they are kept following a reboot. 

This should be just a matter of 
mkdir -p /var/log/journal

Comment 3 Derek Higgins 2015-06-25 10:42:10 UTC
I'm raising the priority on this because to fix this post GA for an already deployed environments would require one of
  o Redeployment of new images
  o Document how to make the journal logs persistent across a deployment,
    for each node:
        mkdir -p /var/log/journal
        systemctl restart systemd-journald

Since to fix this now shouldn't be a large chunk of work I think it makes sense to try and get it in

Comment 4 Mark McLoughlin 2015-06-26 06:22:52 UTC
See also https://access.redhat.com/solutions/696893 "How to enable persistent logging for the systemd journal"

In https://access.redhat.com/articles/1346583 there's a reference to persistent logging being the default for RHEL Atomic Host. See also bug #1189872

Comment 7 Omri Hochman 2015-07-06 19:59:27 UTC
failed_qa : instack-undercloud-2.1.2-11.el7ost.noarch   

couldn't find the folder /var/log/journal - doesn't exist.

Comment 8 James Slagle 2015-07-06 21:34:26 UTC
(In reply to Omri Hochman from comment #7)
> failed_qa : instack-undercloud-2.1.2-11.el7ost.noarch   
> 
> couldn't find the folder /var/log/journal - doesn't exist.

to confirm, this directory didn't exist on the overcloud nodes?

can you link to the images you used?

Comment 9 James Slagle 2015-07-07 00:42:41 UTC
ok i see the problem is now that the script to create this dir was missing a +x. will fix in a new build.

Comment 11 Omri Hochman 2015-07-10 20:27:05 UTC
Verified with : instack-undercloud-2.1.2-17.el7ost.noarch
----------------------------------------------------------

files exist on the overcloud nodes :
[root@overcloud-compute-0 ~]# less /var/log/journal/e6f98fb919f344c6b51d108e1f6995c2/
system.journal     user-1000.journal

Comment 13 errata-xmlrpc 2015-08-05 13:56:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549


Note You need to log in before you can comment on or make changes to this bug.