On Red Hat 7.2 with OSP 7 (Kilo) (ex: openstack-nova-scheduler RPM version 2015.1.2), the Guru Mediation Report doesn't work on nova-scheduler nor nova-api: sending USR1 signal to the main PID writes this message to stderr (which can be seen in "systemctl status openstack-nova-api" for example): "nova-api[5912]: Unable to run Guru Meditation Report!" How reproducible: Always :-/ Steps to Reproduce: 1. start openstack-nova-api service: systemctl start openstack-nova-api 2. wait a few seconds until the service is ready (wait for "active" in "systemctl status openstack-nova-api") 3. get the main PID from "systemctl start openstack-nova-api" and send a SIGUSR1 signal, ex: kill -USR1 12345 4. check the Guru Mediation Report in stderr using: "systemctl status openstack-nova-api" (or try: journalctl -f -u openstack-nova-api, run it before sending the signal, -f is for tailing it) Actual results: "nova-api[5912]: Unable to run Guru Meditation Report!" Expected results: See http://docs.openstack.org/developer/nova/gmr.html & https://www.berrange.com/posts/2015/02/19/nova-and-its-use-of-olso-incubator-guru-meditation-reports/
Worth pointing out that GMR seems to be working fine for nova-compute service
This seems to be an SELinux issue. GMR uses the psutil package to report on all of the children of the process which in turn walks the whole of /proc opening each /proc/PID/status to find all the processes which have ppid equal to that of the running process, and also checks the creation time looking at /proc/$PID/stat. this seems to be what is failing. After setenforce 0 guru meditation seems to work just fine. A typical AVC denial would be: type=AVC msg=audit(1450479143.971:18911): avc: denied { getattr } for pid=4659 comm="nova-scheduler" path="/proc/26813/stat" dev="proc" ino=169648 scontext=system_u:system_r:nova_t:s0 tcontext=system_u:system_r:nrpe_t:s0 tclass=file Moving to the selinux component. Please note that this is a reasonably high priority thing to resolve as GMR is critical to troubleshooting customer deployments.
I will confirm if this is an issue with rhos 8 as well and open another bug if it is.
Have you confirmed this is in rhos 8? This rule is reasonable to add. Make sure you were testing to permissive to be positive this is the only rule we need here. allow nova_t nrpe_t:file getattr;
OK, so: 1) GMR causes walking /proc via calls to python-psutil 2) /proc files can be practically any label on the system: On a non-OpenStack system, I get 29 # find /proc -name 'status' | xargs ls -lZ | \ awk '{print $4}' | sort -u | wc -l 3) GMR is captured by root sending a signal to a process 4) Root is unconfined My initial opinions: 1) GMR is important for debugging 2) To be able to read any file in /proc to determine things, a process calling GMR would need to run unconfined, or be able to read nearly any label on the system. 3) SELinux is supposed to prevent (2), and (2) is a bad idea. Now, debugging a production environment should not be the frequent / normal case we optimize for. Certainly, running with 'setenforce 0' should not be the normal case, either. However, when we optimize SELinux policies for the debug state so debugging tools can have blanket access to read almost every label on the system instead of optimizing for normal (production-running) state where this is not required, we're doing it wrong. So, I read the python-psutil code briefly that GMR uses, and reproduced the problem on a running host: Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: if p.ppid == self.pid: Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: File "/usr/lib64/python2.7/site-packages/psutil/__init__.py", line 335, in ppid Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: return self._platform_impl.get_process_ppid() Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: File "/usr/lib64/python2.7/site-packages/psutil/_pslinux.py", line 471, in wrapper Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: raise AccessDenied(self.pid, self._process_name) Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: AccessDenied: (pid=1) Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: Unable to run Guru Meditation Report! It is probable that proper error handling would be sufficient to make GMR work in most cases without needing to make OpenStack daemons default to (effectively) wide open permissions. Remember - once we allow nova to read 'bind_t' or whatever, it's not restricted to files in /proc if it is compromised. Anyway, it's also possible another binary with specific permissions (and a domain transition) can be built to enable the information gathering required by GMR (though this would decrease the elegance of how GMR works, IMO). I'd first recommend trying to fix python-psutil. Once we have error handling for SELinux confined use cases, if there are specific permissions needed to get a GMR for a given process (invented example: nova_api_t needs to also be able to read nova_scheduler_t or something like that), we can look at them at that point. My conclusion for now: I think you should simply temporarily turn off enforcing when capturing a GMR to debug an issue, and then turn it back on: setenforce 0 killall -USR2 openstack-nova-api killall -USR2 ... sleep 10 setenforce 1