Bug 1292787 - Guru Mediation Report doesn't work on nova-scheduler nor nova-api
Guru Mediation Report doesn't work on nova-scheduler nor nova-api
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
urgent Severity urgent
: ---
: 7.0 (Kilo)
Assigned To: Lon Hohberger
Shai Revivo
: Triaged, ZStream
Depends On:
  Show dependency treegraph
Reported: 2015-12-18 05:51 EST by Victor Stinner
Modified: 2017-02-17 14:09 EST (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1293280 (view as bug list)
Last Closed: 2017-02-17 14:09:51 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Victor Stinner 2015-12-18 05:51:25 EST
On Red Hat 7.2 with OSP 7 (Kilo) (ex: openstack-nova-scheduler RPM version 2015.1.2), the Guru Mediation Report doesn't work on nova-scheduler nor nova-api: sending USR1 signal to the main PID writes this message to stderr (which can be seen in "systemctl status openstack-nova-api" for example):

"nova-api[5912]: Unable to run Guru Meditation Report!"

How reproducible:

Always :-/

Steps to Reproduce:

1. start openstack-nova-api service: systemctl start openstack-nova-api
2. wait a few seconds until the service is ready (wait for "active" in "systemctl status openstack-nova-api")
3. get the main PID from "systemctl start openstack-nova-api" and send a SIGUSR1 signal, ex: kill -USR1 12345
4. check the Guru Mediation Report in stderr using: "systemctl status openstack-nova-api" (or try: journalctl -f -u openstack-nova-api, run it before sending the signal, -f is for tailing it)

Actual results:

"nova-api[5912]: Unable to run Guru Meditation Report!"

Expected results:

See http://docs.openstack.org/developer/nova/gmr.html & https://www.berrange.com/posts/2015/02/19/nova-and-its-use-of-olso-incubator-guru-meditation-reports/
Comment 3 Nikola Dipanov 2015-12-18 05:56:04 EST
Worth pointing out that GMR seems to be working fine for nova-compute service
Comment 4 Nikola Dipanov 2015-12-18 17:55:44 EST
This seems to be an SELinux issue. GMR uses the psutil package to report on all of the children of the process which in turn walks the whole of /proc opening each /proc/PID/status to find all the processes which have ppid equal to that of the running process, and also checks the creation time looking at /proc/$PID/stat.

this seems to be what is failing. After setenforce 0 guru meditation seems to work just fine. A typical AVC denial would be:

type=AVC msg=audit(1450479143.971:18911): avc:  denied  { getattr } for  pid=4659 comm="nova-scheduler" path="/proc/26813/stat" dev="proc" ino=169648 scontext=system_u:system_r:nova_t:s0 tcontext=system_u:system_r:nrpe_t:s0 tclass=file

Moving to the selinux component.

Please note that this is a reasonably high priority thing to resolve as GMR is critical to troubleshooting customer deployments.
Comment 5 Nikola Dipanov 2015-12-18 17:57:19 EST
I will confirm if this is an issue with rhos 8 as well and open another bug if it is.
Comment 6 Ryan Hallisey 2016-01-12 09:23:54 EST
Have you confirmed this is in rhos 8?  This rule is reasonable to add.  Make sure you were testing to permissive to be positive this is the only rule we need here. 

allow nova_t nrpe_t:file getattr;
Comment 10 Lon Hohberger 2017-02-17 14:09:51 EST
OK, so:

1) GMR causes walking /proc via calls to python-psutil

2) /proc files can be practically any label on the system:

  On a non-OpenStack system, I get 29 

  # find /proc -name 'status' |  xargs ls -lZ | \
    awk '{print $4}' | sort -u | wc -l

3) GMR is captured by root sending a signal to a process

4) Root is unconfined

My initial opinions:

1) GMR is important for debugging

2) To be able to read any file in /proc to determine things, a process calling GMR would need to run unconfined, or be able to read nearly any label on the system.

3) SELinux is supposed to prevent (2), and (2) is a bad idea.

Now, debugging a production environment should not be the frequent / normal case we optimize for.  Certainly, running with 'setenforce 0' should not be the normal case, either. However, when we optimize SELinux policies for the debug state so debugging tools can have blanket access to read almost every label on the system instead of optimizing for normal (production-running) state where this is not required, we're doing it wrong.

So, I read the python-psutil code briefly that GMR uses, and reproduced the problem on a running host:

Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: if p.ppid == self.pid:
Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: File "/usr/lib64/python2.7/site-packages/psutil/__init__.py", line 335, in ppid
Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: return self._platform_impl.get_process_ppid()
Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: File "/usr/lib64/python2.7/site-packages/psutil/_pslinux.py", line 471, in wrapper
Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: raise AccessDenied(self.pid, self._process_name)
Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: AccessDenied: (pid=1)
Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: Unable to run Guru Meditation Report!

It is probable that proper error handling would be sufficient to make GMR work in most cases without needing to make OpenStack daemons default to (effectively) wide open permissions. Remember - once we allow nova to read 'bind_t' or whatever, it's not restricted to files in /proc if it is compromised.

Anyway, it's also possible another binary with specific permissions (and a domain transition) can be built to enable the information gathering required by GMR (though this would decrease the elegance of how GMR works, IMO).

I'd first recommend trying to fix python-psutil. Once we have error handling for SELinux confined use cases, if there are specific permissions needed to get a GMR for a given process (invented example: nova_api_t needs to also be able to read nova_scheduler_t or something like that), we can look at them at that point.

My conclusion for now: I think you should simply temporarily turn off enforcing when capturing a GMR to debug an issue, and then turn it back on:

  setenforce 0
  killall -USR2 openstack-nova-api
  killall -USR2 ...
  sleep 10
  setenforce 1

Note You need to log in before you can comment on or make changes to this bug.