Bug 1387347 - Permission denied: '/var/log/aodh/app.log' after upgrade from 9->10
Summary: Permission denied: '/var/log/aodh/app.log' after upgrade from 9->10
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: 10.0 (Newton)
Assignee: Lon Hohberger
QA Contact: Yurii Prokulevych
Depends On:
TreeView+ depends on / blocked
Reported: 2016-10-20 16:41 UTC by Yurii Prokulevych
Modified: 2016-12-14 16:24 UTC (History)
12 users (show)

Fixed In Version: openstack-selinux-0.7.11-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2016-12-14 16:24:10 UTC

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC

Description Yurii Prokulevych 2016-10-20 16:41:59 UTC
Description of problem:
AODH Api fails to reply

aodh alarm list
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
 (HTTP 500)

Excerpt from /var/log/httpd/aodh_wsgi_error.log
[Thu Oct 20 16:20:33.594355 2016] [:error] [pid 19006] [remote] mod_wsgi (pid=19006): Target WSGI script '/var/www/cgi-bin/aodh/app' cannot be loaded as Python module.
[Thu Oct 20 16:20:33.594382 2016] [:error] [pid 19006] [remote] mod_wsgi (pid=19006): Exception occurred processing WSGI script '/var/www/cgi-bin/aodh/app'.
[Thu Oct 20 16:20:33.594422 2016] [:error] [pid 19006] [remote] Traceback (most recent call last):
[Thu Oct 20 16:20:33.594437 2016] [:error] [pid 19006] [remote]   File "/var/www/cgi-bin/aodh/app", line 23, in <module>
[Thu Oct 20 16:20:33.594457 2016] [:error] [pid 19006] [remote]     application = app.build_wsgi_app(argv=[])
[Thu Oct 20 16:20:33.594464 2016] [:error] [pid 19006] [remote]   File "/usr/lib/python2.7/site-packages/aodh/api/app.py", line 79, in build_wsgi_app
[Thu Oct 20 16:20:33.594475 2016] [:error] [pid 19006] [remote]     return load_app(service.prepare_service(argv=argv))
[Thu Oct 20 16:20:33.594481 2016] [:error] [pid 19006] [remote]   File "/usr/lib/python2.7/site-packages/aodh/service.py", line 86, in prepare_service
[Thu Oct 20 16:20:33.594490 2016] [:error] [pid 19006] [remote]     log.setup(conf, 'aodh')
[Thu Oct 20 16:20:33.594496 2016] [:error] [pid 19006] [remote]   File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 269, in setup
[Thu Oct 20 16:20:33.594505 2016] [:error] [pid 19006] [remote]     _setup_logging_from_conf(conf, product_name, version)
[Thu Oct 20 16:20:33.594511 2016] [:error] [pid 19006] [remote]   File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 366, in _setup_logging_from_conf
[Thu Oct 20 16:20:33.594519 2016] [:error] [pid 19006] [remote]     filelog = file_handler(logpath)
[Thu Oct 20 16:20:33.594525 2016] [:error] [pid 19006] [remote]   File "/usr/lib64/python2.7/logging/handlers.py", line 392, in __init__
[Thu Oct 20 16:20:33.594536 2016] [:error] [pid 19006] [remote]     logging.FileHandler.__init__(self, filename, mode, encoding, delay)
[Thu Oct 20 16:20:33.594541 2016] [:error] [pid 19006] [remote]   File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
[Thu Oct 20 16:20:33.594550 2016] [:error] [pid 19006] [remote]     StreamHandler.__init__(self, self._open())
[Thu Oct 20 16:20:33.594555 2016] [:error] [pid 19006] [remote]   File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
[Thu Oct 20 16:20:33.594563 2016] [:error] [pid 19006] [remote]     stream = open(self.baseFilename, self.mode)
[Thu Oct 20 16:20:33.594578 2016] [:error] [pid 19006] [remote] IOError: [Errno 13] Permission denied: '/var/log/aodh/app.log'

ls -lZ /var/log/aodh/
-rw-r--r--. aodh aodh system_u:object_r:var_log_t:s0   aodh-dbsync.log
-rw-r--r--. aodh aodh system_u:object_r:var_log_t:s0   app.log
-rw-r--r--. aodh aodh system_u:object_r:var_log_t:s0   evaluator.log
-rw-r--r--. aodh aodh system_u:object_r:var_log_t:s0   listener.log
-rw-r--r--. aodh aodh system_u:object_r:var_log_t:s0   notifier.log

Might be SELinux related:
time->Thu Oct 20 16:24:32 2016
type=SYSCALL msg=audit(1476980672.939:24030): arch=c000003e syscall=2 success=no exit=-13 a0=7fade1bded20 a1=441 a2=1b6 a3=24 items=0 ppid=662 pid=760 auid=4294967295 uid=993 gid=991 euid=993 suid=993 fsuid=993 egid=991 sgid=991 fsgid=991 tty=(none) ses=4294967295 comm="/usr/sbin/httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1476980672.939:24030): avc:  denied  { open } for  pid=760 comm="/usr/sbin/httpd" path="/var/log/aodh/app.log" dev="vda2" ino=42517 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:var_log_t:s0 tclass=file

Version-Release number of selected component (if applicable):


Steps to Reproduce:
1. Upgrade from rhos-9 on rhel-7.2 to rhos-10 on rhel-7.3
2. Try to list alarms:
    source overcloudrc
    aodh alarm list

Actual results:

Expected results:
AODH returns list of alarms

Additional info:
I've rebooted one controller but this didn't help.

Setup: 3controllers + 1compute + 1ceph

Comment 1 Mike Burns 2016-10-21 11:59:03 UTC
If you run in permissive mode, does that resolve the problem?

If yes, please provide audit.log from a run in permissive mode.


Comment 2 Yurii Prokulevych 2016-10-21 13:52:22 UTC
Setting selinux to permissive resolves issues.

Comment 6 Lon Hohberger 2016-10-25 17:36:05 UTC
I'll look in to this today.

Comment 7 Lon Hohberger 2016-10-25 18:05:05 UTC
Can you try:
  semanage fcontext -a -t httpd_log_t /var/log/aodh/app.log

... and see if that works?  If so, I'll add it to openstack-selinux for now until there's a better policy for aodh.

Comment 8 Yurii Prokulevych 2016-10-31 08:54:24 UTC
Changing the type helps.

But it's also needed to set same type for /var/log/gnocchi/app.log and /var/log/ceilometer/app.log

-rw-r--r--. gnocchi gnocchi system_u:object_r:httpd_log_t:s0 /var/log/gnocchi/app.log
-rw-r--r--. aodh aodh       system_u:object_r:httpd_log_t:s0 /var/log/aodh/app.log
-rw-r--r--. ceilometer ceilometer system_u:object_r:httpd_log_t:s0 /var/log/ceilometer/app.log

Comment 9 Lon Hohberger 2016-10-31 13:31:39 UTC
That actually makes sense.  Anything being served by WSGI via httpd is going to need this type.

Thanks, Yurii

Comment 11 Mike Burns 2016-11-16 00:21:39 UTC
Sasha, is this something you can verify easily?

Comment 12 Yurii Prokulevych 2016-11-16 09:16:26 UTC
Verified with openstack-selinux-0.7.12-1.el7ost.noarch

semanage fcontext -l | grep -E 'aodh|ceilometer|gnocchi'
/var/log/gnocchi/app.log                           all files          system_u:object_r:httpd_log_t:s0 
/var/log/aodh/app.log                              all files          system_u:object_r:httpd_log_t:s0 
/var/log/ceilometer/app.log                        all files          system_u:object_r:httpd_log_t:s0 

ls -lZ /var/log/aodh/app.log 
-rw-r--r--. aodh aodh system_u:object_r:httpd_log_t:s0 /var/log/aodh/app.log

ls -lZ /var/log/ceilometer/app.log 
-rw-r--r--. ceilometer ceilometer system_u:object_r:httpd_log_t:s0 /var/log/ceilometer/app.log

ls -lZ /var/log/gnocchi/app.log 
-rw-r--r--. gnocchi gnocchi system_u:object_r:httpd_log_t:s0 /var/log/gnocchi/app.log

Comment 14 errata-xmlrpc 2016-12-14 16:24:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.