Bug 1249751 - centralized logging - fluentd is blocked from accessing Nova compute logs
Summary: centralized logging - fluentd is blocked from accessing Nova compute logs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z3
: 7.0 (Kilo)
Assignee: Artom Lifshitz
QA Contact: Gabriel Szasz
URL:
Whiteboard:
Depends On:
Blocks: 1284659
TreeView+ depends on / blocked
 
Reported: 2015-08-03 17:28 UTC by Ben England
Modified: 2019-09-09 14:49 UTC (History)
20 users (show)

Fixed In Version: openstack-nova-2015.1.2-5.el7ost
Doc Type: Bug Fix
Doc Text:
The group ownership of the nova log directory has been changed from "root" to "nova", which will allow fluentd to access the logs.
Clone Of:
: 1284659 (view as bug list)
Environment:
Last Closed: 2015-12-21 17:05:32 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:2673 normal SHIPPED_LIVE Moderate: openstack-nova security and bug fix advisory 2015-12-21 21:51:35 UTC

Description Ben England 2015-08-03 17:28:52 UTC
NOTE: you should reassign component, I could not find right component to log bz against.

Description of problem:

why this matters: centralized logging is critical for management of large RHEL OSP configs.  fluentd ( http://www.fluentd.org/ ) is part of this solution.

fluentd cannot access openstack logs because the owning group of the logfile directory is "root" not the user associated with the service.  In RHEL OSP 6 I typically see /var/log/{nova,cinder}, but there are others.

[root@gprfc041 fluentd(keystone_admin)]# ls -ld /var/log/* | grep '^d' | grep -v 'root       root'
drwxr-x---. 2 cinder     root           4096 Jul  1 10:42 /var/log/cinder
drwxr-x---. 2 nova       root           4096 Jul 30 03:36 /var/log/nova

What I want to see is:

drwxr-x---. 2 cinder     cinder         4096 Jul  1 14:42 /var/log/cinder
drwxr-x---. 2 nova       nova           4096 Jul 30 07:36 /var/log/nova

This results in "Permission denied" errors.

[root@gprfc041 fluentd(keystone_admin)]# strace -f -p 30659 -e open
Process 30659 attached with 21 threads
[pid 30669] open("/var/log/nova/nova-compute.log", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
[pid 30671] open("/var/log/nova/nova-cert.log", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
...

But fluentd is a member of these groups so if above problem is fixed then it can read the log files.

[root@gprfc041 fluentd(keystone_admin)]# groups fluentd
fluentd : fluentd keystone cinder glance nova neutron


Version-Release number of selected component (if applicable):

RHEL OSP 6 
baseurl=http://download.lab.bos.redhat.com/rel-eng/OpenStack/6.0-RHEL-7/latest/RH7-RHOS-6.0/$basearch/os

How reproducible:

every time.

Steps to Reproduce:

0. do usual RHEL OSP install.

1. install fluentd packages we have available at

http://ayanami.boston.devel.redhat.com/poodles/optools/7.0/latest/RHOS-7/x86_64/os/

And follow the documentation at

http://file.bne.redhat.com/~ggillies/optools_doc/

2. try to look at logs with kibana.  For example:

http://ki-perf44.perf.lab.eng.bos.redhat.com

and look at host gprfc041.

Actual results:

July 31st 2015, 21:00:00.000 	

host:
    gprfc045
host.raw:
    gprfc045
ident:
    fluentd
message:
    2015-08-01 01:00:00 -0400 [error]: Permission denied - /var/log/nova/nova-compute.log
@timestamp:
    July 31st 2015, 21:00:00.000
_source:
    {"host":"gprfc045","ident":"fluentd","message":"2015-08-01 01:00:00 -0400 [error]: Permission denied - /var/log/nova/nova-compute.log","@timestamp":"2015-08-01T01:00:00+00:00"}
_id:
    AU7noZoi3xvEGagCvCHu
_type:
    fluentd
_index:
    logstash-2015.08.01 

Expected results:

contents of this log file.

Additional info:

Comment 3 Ben England 2015-08-04 12:29:44 UTC
In RHEL OSP 7, problem is fixed for Nova but not for Cinder and some other services.

[root@overcloud-compute-0 ~]# ls -ld /var/log/* | grep '^d' | grep root | grep -v 'root       root'
drwxr-xr-x. 2 ceilometer root                  81 Aug  2 03:42 /var/log/ceilometer
drwxr-x---. 2 cinder     root                   6 Jun 23 05:51 /var/log/cinder
drwxr-x---. 2 heat       root                   6 Jun 19 15:10 /var/log/heat
drwxr-x---. 2 mongodb    root                   6 May  6 15:45 /var/log/mongodb

Comment 4 Artom Lifshitz 2015-11-23 19:40:28 UTC
Based on the 'Installing the log collection agent on all nodes' section from http://file.bne.redhat.com/~ggillies/optools_doc/ (same link as in Ben's description), the following components may be affected:

nova neutron keystone glance cinder

I checked the %files section in their respective .spec files (since %files has the final say on permissions, and not %install), and found that only Nova and Cinder have their log directory group-owned by root:

./openstack-keystone/openstack-keystone.spec:%dir %attr(0750, keystone, keystone) %{_localstatedir}/log/keystone
./openstack-cinder/openstack-cinder.spec:%dir %attr(0750, cinder, root) %{_localstatedir}/log/cinder
./openstack-neutron/openstack-neutron.spec:%dir %attr(0750, %{service}, %{service}) %{_localstatedir}/log/%{service}
./openstack-glance/openstack-glance.spec:%dir %attr(0750, glance, glance) %{_localstatedir}/log/glance
./openstack-nova/openstack-nova.spec:%dir %attr(0750, nova, root) %{_localstatedir}/log/nova

Once I have qa_ack I'll push a fix for Nova - in the meantime I'm cloning this to Cinder.

Comment 8 Gabriel Szasz 2015-12-15 15:25:11 UTC
Tested against openstack-nova-2015.1.2-7.el7ost

ls -ld /var/log/nova
drwxr-x---. 2 nova nova 4096 Dec 15 09:25 /var/log/nova

The patch was applied and group ownership of the /var/log/nova is fixed and fluentd can now access nova logs.

Note that this issue is still not fixed for cinder (see bug 1284659):

ls -ld /var/log/* | grep '^d' | grep root | grep -v 'root       root'
drwxr-xr-x. 2 ceilometer root       4096 Dec 15 09:35 /var/log/ceilometer
drwxr-x---. 2 cinder     root         98 Dec 15 09:21 /var/log/cinder
drwxr-x---. 2 mongodb    root         24 Dec 15 09:34 /var/log/mongodb

Comment 10 errata-xmlrpc 2015-12-21 17:05:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2673


Note You need to log in before you can comment on or make changes to this bug.