Hide Forgot
Description of problem: ----------------------- AODH Api fails to reply aodh alarm list <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> (HTTP 500) Excerpt from /var/log/httpd/aodh_wsgi_error.log ----------------------------------------------- [Thu Oct 20 16:20:33.594355 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] mod_wsgi (pid=19006): Target WSGI script '/var/www/cgi-bin/aodh/app' cannot be loaded as Python module. [Thu Oct 20 16:20:33.594382 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] mod_wsgi (pid=19006): Exception occurred processing WSGI script '/var/www/cgi-bin/aodh/app'. [Thu Oct 20 16:20:33.594422 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] Traceback (most recent call last): [Thu Oct 20 16:20:33.594437 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] File "/var/www/cgi-bin/aodh/app", line 23, in <module> [Thu Oct 20 16:20:33.594457 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] application = app.build_wsgi_app(argv=[]) [Thu Oct 20 16:20:33.594464 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] File "/usr/lib/python2.7/site-packages/aodh/api/app.py", line 79, in build_wsgi_app [Thu Oct 20 16:20:33.594475 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] return load_app(service.prepare_service(argv=argv)) [Thu Oct 20 16:20:33.594481 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] File "/usr/lib/python2.7/site-packages/aodh/service.py", line 86, in prepare_service [Thu Oct 20 16:20:33.594490 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] log.setup(conf, 'aodh') [Thu Oct 20 16:20:33.594496 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 269, in setup [Thu Oct 20 16:20:33.594505 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] _setup_logging_from_conf(conf, product_name, version) [Thu Oct 20 16:20:33.594511 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 366, in _setup_logging_from_conf [Thu Oct 20 16:20:33.594519 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] filelog = file_handler(logpath) [Thu Oct 20 16:20:33.594525 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] File "/usr/lib64/python2.7/logging/handlers.py", line 392, in __init__ [Thu Oct 20 16:20:33.594536 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] logging.FileHandler.__init__(self, filename, mode, encoding, delay) [Thu Oct 20 16:20:33.594541 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__ [Thu Oct 20 16:20:33.594550 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] StreamHandler.__init__(self, self._open()) [Thu Oct 20 16:20:33.594555 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open [Thu Oct 20 16:20:33.594563 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] stream = open(self.baseFilename, self.mode) [Thu Oct 20 16:20:33.594578 2016] [:error] [pid 19006] [remote 172.17.1.15:21230] IOError: [Errno 13] Permission denied: '/var/log/aodh/app.log' ls -lZ /var/log/aodh/ -rw-r--r--. aodh aodh system_u:object_r:var_log_t:s0 aodh-dbsync.log -rw-r--r--. aodh aodh system_u:object_r:var_log_t:s0 app.log -rw-r--r--. aodh aodh system_u:object_r:var_log_t:s0 evaluator.log -rw-r--r--. aodh aodh system_u:object_r:var_log_t:s0 listener.log -rw-r--r--. aodh aodh system_u:object_r:var_log_t:s0 notifier.log Might be SELinux related: ------------------------- time->Thu Oct 20 16:24:32 2016 type=SYSCALL msg=audit(1476980672.939:24030): arch=c000003e syscall=2 success=no exit=-13 a0=7fade1bded20 a1=441 a2=1b6 a3=24 items=0 ppid=662 pid=760 auid=4294967295 uid=993 gid=991 euid=993 suid=993 fsuid=993 egid=991 sgid=991 fsgid=991 tty=(none) ses=4294967295 comm="/usr/sbin/httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null) type=AVC msg=audit(1476980672.939:24030): avc: denied { open } for pid=760 comm="/usr/sbin/httpd" path="/var/log/aodh/app.log" dev="vda2" ino=42517 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:var_log_t:s0 tclass=file Version-Release number of selected component (if applicable): ------------------------------------------------------------- openstack-aodh-notifier-3.0.0-1.el7ost.noarch puppet-aodh-9.4.0-1.el7ost.noarch python-aodh-3.0.0-1.el7ost.noarch python-aodhclient-0.7.0-1.el7ost.noarch openstack-aodh-common-3.0.0-1.el7ost.noarch openstack-aodh-api-3.0.0-1.el7ost.noarch openstack-aodh-listener-3.0.0-1.el7ost.noarch openstack-aodh-evaluator-3.0.0-1.el7ost.noarch libselinux-ruby-2.5-6.el7.x86_64 libselinux-utils-2.5-6.el7.x86_64 selinux-policy-targeted-3.13.1-102.el7.noarch libselinux-2.5-6.el7.x86_64 ceph-selinux-10.2.2-41.el7cp.x86_64 openstack-selinux-0.7.11-1.el7ost.noarch libselinux-python-2.5-6.el7.x86_64 selinux-policy-3.13.1-102.el7.noarch Steps to Reproduce: ------------------- 1. Upgrade from rhos-9 on rhel-7.2 to rhos-10 on rhel-7.3 2. Try to list alarms: source overcloudrc aodh alarm list Actual results: --------------- Error Expected results: ----------------- AODH returns list of alarms Additional info: ---------------- I've rebooted one controller but this didn't help. Setup: 3controllers + 1compute + 1ceph
If you run in permissive mode, does that resolve the problem? If yes, please provide audit.log from a run in permissive mode. Thanks
Setting selinux to permissive resolves issues.
I'll look in to this today.
Can you try: semanage fcontext -a -t httpd_log_t /var/log/aodh/app.log ... and see if that works? If so, I'll add it to openstack-selinux for now until there's a better policy for aodh.
Changing the type helps. But it's also needed to set same type for /var/log/gnocchi/app.log and /var/log/ceilometer/app.log -rw-r--r--. gnocchi gnocchi system_u:object_r:httpd_log_t:s0 /var/log/gnocchi/app.log -rw-r--r--. aodh aodh system_u:object_r:httpd_log_t:s0 /var/log/aodh/app.log -rw-r--r--. ceilometer ceilometer system_u:object_r:httpd_log_t:s0 /var/log/ceilometer/app.log
That actually makes sense. Anything being served by WSGI via httpd is going to need this type. Thanks, Yurii
Sasha, is this something you can verify easily?
Verified with openstack-selinux-0.7.12-1.el7ost.noarch semanage fcontext -l | grep -E 'aodh|ceilometer|gnocchi' /var/log/gnocchi/app.log all files system_u:object_r:httpd_log_t:s0 /var/log/aodh/app.log all files system_u:object_r:httpd_log_t:s0 /var/log/ceilometer/app.log all files system_u:object_r:httpd_log_t:s0 ls -lZ /var/log/aodh/app.log -rw-r--r--. aodh aodh system_u:object_r:httpd_log_t:s0 /var/log/aodh/app.log ls -lZ /var/log/ceilometer/app.log -rw-r--r--. ceilometer ceilometer system_u:object_r:httpd_log_t:s0 /var/log/ceilometer/app.log ls -lZ /var/log/gnocchi/app.log -rw-r--r--. gnocchi gnocchi system_u:object_r:httpd_log_t:s0 /var/log/gnocchi/app.log
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html