Bug 1292787

Worth pointing out that GMR seems to be working fine for nova-compute service

This seems to be an SELinux issue. GMR uses the psutil package to report on all of the children of the process which in turn walks the whole of /proc opening each /proc/PID/status to find all the processes which have ppid equal to that of the running process, and also checks the creation time looking at /proc/$PID/stat.

this seems to be what is failing. After setenforce 0 guru meditation seems to work just fine. A typical AVC denial would be:

type=AVC msg=audit(1450479143.971:18911): avc:  denied  { getattr } for  pid=4659 comm="nova-scheduler" path="/proc/26813/stat" dev="proc" ino=169648 scontext=system_u:system_r:nova_t:s0 tcontext=system_u:system_r:nrpe_t:s0 tclass=file

Moving to the selinux component.

Please note that this is a reasonably high priority thing to resolve as GMR is critical to troubleshooting customer deployments.

I will confirm if this is an issue with rhos 8 as well and open another bug if it is.

Have you confirmed this is in rhos 8?  This rule is reasonable to add.  Make sure you were testing to permissive to be positive this is the only rule we need here. 

allow nova_t nrpe_t:file getattr;

OK, so:

1) GMR causes walking /proc via calls to python-psutil

2) /proc files can be practically any label on the system:

  On a non-OpenStack system, I get 29 

  # find /proc -name 'status' |  xargs ls -lZ | \
    awk '{print $4}' | sort -u | wc -l

3) GMR is captured by root sending a signal to a process

4) Root is unconfined


My initial opinions:

1) GMR is important for debugging

2) To be able to read any file in /proc to determine things, a process calling GMR would need to run unconfined, or be able to read nearly any label on the system.

3) SELinux is supposed to prevent (2), and (2) is a bad idea.


Now, debugging a production environment should not be the frequent / normal case we optimize for.  Certainly, running with 'setenforce 0' should not be the normal case, either. However, when we optimize SELinux policies for the debug state so debugging tools can have blanket access to read almost every label on the system instead of optimizing for normal (production-running) state where this is not required, we're doing it wrong.


So, I read the python-psutil code briefly that GMR uses, and reproduced the problem on a running host:

Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: if p.ppid == self.pid:
Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: File "/usr/lib64/python2.7/site-packages/psutil/__init__.py", line 335, in ppid
Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: return self._platform_impl.get_process_ppid()
Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: File "/usr/lib64/python2.7/site-packages/psutil/_pslinux.py", line 471, in wrapper
Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: raise AccessDenied(self.pid, self._process_name)
Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: AccessDenied: (pid=1)
Feb 17 14:00:12 localhost.localdomain nova-scheduler[17473]: Unable to run Guru Meditation Report!

It is probable that proper error handling would be sufficient to make GMR work in most cases without needing to make OpenStack daemons default to (effectively) wide open permissions. Remember - once we allow nova to read 'bind_t' or whatever, it's not restricted to files in /proc if it is compromised.

Anyway, it's also possible another binary with specific permissions (and a domain transition) can be built to enable the information gathering required by GMR (though this would decrease the elegance of how GMR works, IMO).

I'd first recommend trying to fix python-psutil. Once we have error handling for SELinux confined use cases, if there are specific permissions needed to get a GMR for a given process (invented example: nova_api_t needs to also be able to read nova_scheduler_t or something like that), we can look at them at that point.

My conclusion for now: I think you should simply temporarily turn off enforcing when capturing a GMR to debug an issue, and then turn it back on:

  setenforce 0
  killall -USR2 openstack-nova-api
  killall -USR2 ...
  sleep 10
  setenforce 1