Red Hat Bugzilla – Bug 1249751
centralized logging - fluentd is blocked from accessing Nova compute logs
Last modified: 2015-12-21 12:05:32 EST
NOTE: you should reassign component, I could not find right component to log bz against. Description of problem: why this matters: centralized logging is critical for management of large RHEL OSP configs. fluentd ( http://www.fluentd.org/ ) is part of this solution. fluentd cannot access openstack logs because the owning group of the logfile directory is "root" not the user associated with the service. In RHEL OSP 6 I typically see /var/log/{nova,cinder}, but there are others. [root@gprfc041 fluentd(keystone_admin)]# ls -ld /var/log/* | grep '^d' | grep -v 'root root' drwxr-x---. 2 cinder root 4096 Jul 1 10:42 /var/log/cinder drwxr-x---. 2 nova root 4096 Jul 30 03:36 /var/log/nova What I want to see is: drwxr-x---. 2 cinder cinder 4096 Jul 1 14:42 /var/log/cinder drwxr-x---. 2 nova nova 4096 Jul 30 07:36 /var/log/nova This results in "Permission denied" errors. [root@gprfc041 fluentd(keystone_admin)]# strace -f -p 30659 -e open Process 30659 attached with 21 threads [pid 30669] open("/var/log/nova/nova-compute.log", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) [pid 30671] open("/var/log/nova/nova-cert.log", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) ... But fluentd is a member of these groups so if above problem is fixed then it can read the log files. [root@gprfc041 fluentd(keystone_admin)]# groups fluentd fluentd : fluentd keystone cinder glance nova neutron Version-Release number of selected component (if applicable): RHEL OSP 6 baseurl=http://download.lab.bos.redhat.com/rel-eng/OpenStack/6.0-RHEL-7/latest/RH7-RHOS-6.0/$basearch/os How reproducible: every time. Steps to Reproduce: 0. do usual RHEL OSP install. 1. install fluentd packages we have available at http://ayanami.boston.devel.redhat.com/poodles/optools/7.0/latest/RHOS-7/x86_64/os/ And follow the documentation at http://file.bne.redhat.com/~ggillies/optools_doc/ 2. try to look at logs with kibana. For example: http://ki-perf44.perf.lab.eng.bos.redhat.com and look at host gprfc041. Actual results: July 31st 2015, 21:00:00.000 host: gprfc045 host.raw: gprfc045 ident: fluentd message: 2015-08-01 01:00:00 -0400 [error]: Permission denied - /var/log/nova/nova-compute.log @timestamp: July 31st 2015, 21:00:00.000 _source: {"host":"gprfc045","ident":"fluentd","message":"2015-08-01 01:00:00 -0400 [error]: Permission denied - /var/log/nova/nova-compute.log","@timestamp":"2015-08-01T01:00:00+00:00"} _id: AU7noZoi3xvEGagCvCHu _type: fluentd _index: logstash-2015.08.01 Expected results: contents of this log file. Additional info:
In RHEL OSP 7, problem is fixed for Nova but not for Cinder and some other services. [root@overcloud-compute-0 ~]# ls -ld /var/log/* | grep '^d' | grep root | grep -v 'root root' drwxr-xr-x. 2 ceilometer root 81 Aug 2 03:42 /var/log/ceilometer drwxr-x---. 2 cinder root 6 Jun 23 05:51 /var/log/cinder drwxr-x---. 2 heat root 6 Jun 19 15:10 /var/log/heat drwxr-x---. 2 mongodb root 6 May 6 15:45 /var/log/mongodb
Based on the 'Installing the log collection agent on all nodes' section from http://file.bne.redhat.com/~ggillies/optools_doc/ (same link as in Ben's description), the following components may be affected: nova neutron keystone glance cinder I checked the %files section in their respective .spec files (since %files has the final say on permissions, and not %install), and found that only Nova and Cinder have their log directory group-owned by root: ./openstack-keystone/openstack-keystone.spec:%dir %attr(0750, keystone, keystone) %{_localstatedir}/log/keystone ./openstack-cinder/openstack-cinder.spec:%dir %attr(0750, cinder, root) %{_localstatedir}/log/cinder ./openstack-neutron/openstack-neutron.spec:%dir %attr(0750, %{service}, %{service}) %{_localstatedir}/log/%{service} ./openstack-glance/openstack-glance.spec:%dir %attr(0750, glance, glance) %{_localstatedir}/log/glance ./openstack-nova/openstack-nova.spec:%dir %attr(0750, nova, root) %{_localstatedir}/log/nova Once I have qa_ack I'll push a fix for Nova - in the meantime I'm cloning this to Cinder.
Tested against openstack-nova-2015.1.2-7.el7ost ls -ld /var/log/nova drwxr-x---. 2 nova nova 4096 Dec 15 09:25 /var/log/nova The patch was applied and group ownership of the /var/log/nova is fixed and fluentd can now access nova logs. Note that this issue is still not fixed for cinder (see bug 1284659): ls -ld /var/log/* | grep '^d' | grep root | grep -v 'root root' drwxr-xr-x. 2 ceilometer root 4096 Dec 15 09:35 /var/log/ceilometer drwxr-x---. 2 cinder root 98 Dec 15 09:21 /var/log/cinder drwxr-x---. 2 mongodb root 24 Dec 15 09:34 /var/log/mongodb
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2015:2673