| Summary: | cloudinit fails to install chef because of selinux problem. | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Jeremy <jmelvin> |
| Component: | cloud-init | Assignee: | Lars Kellogg-Stedman <lars> |
| Status: | CLOSED DUPLICATE | QA Contact: | |
| Severity: | urgent | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.1 | CC: | dwalsh, jmelvin, lars, lvrabec, mgrepl, mmalik, plautrba, pvrabec, rkharwar, sasha, ssekidde |
| Target Milestone: | rc | Keywords: | Reopened |
| Target Release: | 7.1 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
A bug in libselinux prevented cloud-init from creating the chef runtime state directory in /var/run/chef (when /var/run is a symlink to /run). This works around the issue by explicitly using the path '/run/chef', which avoids the libselinux bug.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-01-13 07:51:00 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
|
Description
Jeremy
2016-11-17 21:54:10 UTC
RHEL-7.1 ======== # rpm -q libselinux libselinux-2.2.2-6.el7.x86_64 # python Python 2.7.5 (default, Feb 11 2014, 07:46:25) [GCC 4.8.2 20140120 (Red Hat 4.8.2-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import selinux >>> selinux.restorecon('/var/run/chef') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 84, in restorecon mode = os.lstat(path)[stat.ST_MODE] OSError: [Errno 2] No such file or directory: '/run/chef' >>> # RHEL-7.3 ======== # rpm -q libselinux libselinux-2.5-6.el7.x86_64 # python Python 2.7.5 (default, Aug 2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import selinux >>> selinux.restorecon('/var/run/chef') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 84, in restorecon mode = os.lstat(path)[stat.ST_MODE] OSError: [Errno 2] No such file or directory: '/run/chef' >>> # > selinux.restorecon(‘/var/run/chef’) ^^^^^^^^^ > OSError: [Errno 2] No such file or directory: '/run/chef' ^^^^^ Is this a problem is libselinux? The /var/run symlink is apparently followed and it looks like /run/chef doesn't exist. When it exists, restorecon works as expected: >>> selinux.restorecon('/var/run/chef') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 84, in restorecon mode = os.lstat(path)[stat.ST_MODE] OSError: [Errno 2] No such file or directory: '/run/chef' >>> os.mkdir("/run/chef") >>> selinux.restorecon('/var/run/chef') >>> Here is the output of ls -l /var/run along with some more background. Can you try to reproduce the error using the same steps. It looks like other users have reported the same issue at https://bugs.launchpad.net/cloud-init/+bug/1385860 . [root@hostname ~]# ls -l /var/run lrwxrwxrwx. 1 root root 6 Aug 1 14:25 /var/run -> ../run ======== Background ========== We are trying to create an instance of Red Hat Enterprise Linux Server release 7.2 (Maipo) on Red Hat Openstack Platform 8. The image has selinux enabled in permissive mode. When we send a request to create a new instance, we provide a cloud-init file which includes a section to install and run chef-client using the "omnibus" install_package as described at http://cloudinit.readthedocs.io/en/latest/topics/examples.html#install-and-run-chef-recipes . Here’s an example of the chef section, with sensitive data replaced with example data. Example chef config: #cloud-config --- chef: environment: example-environment exec: true force_install: false install_type: omnibus log_level: info node_name: ‘example12345’ omnibus_url: http://c2c-repo1-d.fmr.com/chef/omnibus/install.sh server_url: https://c2cchefbosbe.fmr.com/organizations/c2c-int-dev validation_key: '**key**' ====== Problem ================= During instance creation cloud-init fails on the chef section when cloud-init sets up the required directories. Here’s an example stack trace. We can reproduce this on demand using the steps below. Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: In ensure_dir path: /var/run/chef Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: PATH: /run Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: Restoring selinux mode for /run (recursive=True) Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: 2016-11-29 21:04:37,208 - util.py[WARNING]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[WARNING]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.pyc'> Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.pyc'>) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cloudinit/stages.py", line 658, in _run_modules cc.run(run_name, mod.handle, func_args, freq=freq) File "/usr/lib/python2.7/site-packages/cloudinit/cloud.py", line 63, in run return self._runners.run(name, functor, args, freq, clear_on_fail) File "/usr/lib/python2.7/site-packages/cloudinit/helpers.py", line 197, in run results = functor(*args) File "/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.py", line 53, in handle util.ensure_dir(d) File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1291, in ensure_dir os.makedirs(path) File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 166, in __exit__ self.selinux.restorecon(path, recursive=self.recursive) File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 95, in restorecon for fname in fnames]), None) File "/usr/lib64/python2.7/posixpath.py", line 238, in walk func(arg, top, names) File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 95, in <lambda> for fname in fnames]), None) File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 85, in restorecon status, context = matchpathcon(path, mode) OSError: [Errno 2] No such file or directory ===== Steps to Reproduce ========== Create an instance of Red Hat Enterprise Linux Server release 7.2 (Maipo) with selinux enabled in permissive mode on RHOSP 8 with a cloud-init file that installs chef using install it should be creating "/var/run/chef" directory as part of it's operation, so that not existing shouldn't be the issue here.
In cloud-init's cc_chef.py:
<snip>
CHEF_DIRS = [
'/etc/chef',
'/var/log/chef',
'/var/lib/chef',
'/var/cache/chef',
'/var/backups/chef',
'/var/run/chef',
]
<snip>
# Ensure the chef directories we use exist
for d in CHEF_DIRS:
util.ensure_dir(d)
<snip>
cloudinit/util.py:
<snip>
def ensure_dir(path, mode=None):
if not os.path.isdir(path):
# Make the dir and adjust the mode
with SeLinuxGuard(os.path.dirname(path), recursive=True):
os.makedirs(path)
chmod(path, mode)
else:
# Just adjust the mode
chmod(path, mode)
<snip>
class SeLinuxGuard(object):
def __init__(self, path, recursive=False):
# Late import since it might not always
# be possible to use this
try:
self.selinux = importer.import_module('selinux')
except ImportError:
self.selinux = None
self.path = path
self.recursive = recursive
def __enter__(self):
if self.selinux and self.selinux.is_selinux_enabled():
return True
else:
return False
def __exit__(self, excp_type, excp_value, excp_traceback):
if not self.selinux or not self.selinux.is_selinux_enabled():
return
if not os.path.lexists(self.path):
return
path = os.path.realpath(self.path)
# path should be a string, not unicode
path = str(path)
try:
stats = os.lstat(path)
self.selinux.matchpathcon(path, stats[stat.ST_MODE])
except OSError:
return
LOG.debug("Restoring selinux mode for %s (recursive=%s)",
path, self.recursive)
self.selinux.restorecon(path, recursive=self.recursive)
<snip>
So, we're calling 'restorecon' from the selinux module:
site-packages/selinux/__init__.py
<snip>
def restorecon(path, recursive=False):
""" Restore SELinux context on a given path """
try:
mode = os.lstat(path)[stat.ST_MODE]
status, context = matchpathcon(path, mode)
except OSError:
path = os.path.realpath(os.path.expanduser(path))
mode = os.lstat(path)[stat.ST_MODE]
status, context = matchpathcon(path, mode)
...
<snip>
That's where we're failing per the error log:
<error>
Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: In ensure_dir path: /var/run/chef
Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: PATH: /run
Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: Restoring selinux mode for /run (recursive=True)
Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: 2016-11-29 21:04:37,208 - util.py[WARNING]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config
Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[WARNING]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.pyc'>
Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.pyc'>)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cloudinit/stages.py", line 658, in _run_modules
cc.run(run_name, mod.handle, func_args, freq=freq)
File "/usr/lib/python2.7/site-packages/cloudinit/cloud.py", line 63, in run
return self._runners.run(name, functor, args, freq, clear_on_fail)
File "/usr/lib/python2.7/site-packages/cloudinit/helpers.py", line 197, in run
results = functor(*args)
File "/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.py", line 53, in handle
util.ensure_dir(d)
File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1291, in ensure_dir
os.makedirs(path)
File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 166, in __exit__
self.selinux.restorecon(path, recursive=self.recursive)
File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 95, in restorecon
for fname in fnames]), None)
File "/usr/lib64/python2.7/posixpath.py", line 238, in walk
func(arg, top, names)
File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 95, in <lambda>
for fname in fnames]), None)
File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 85, in restorecon
status, context = matchpathcon(path, mode)
OSError: [Errno 2] No such file or directory
</snip>
Again, it's not immediately clear why 'matchpathcon' would be having issue here. There were a few fixes earlier in RHEL's life to allow it to handle relative paths and the like, but I don't see anything outstanding that's interesting. I'm still not quite clear what it's trying that's not available to it at runtime, so we need to do a bit more digging here.
[root@shale foo]# chcon system_u:object_r:var_run_t mega [root@shale foo]# ls -lZ mega -rw-r--r--. root root system_u:object_r:var_run_t mega [root@shale foo]# python Python 2.7.5 (default, Aug 2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import selinux >>> selinux.restorecon('/root/foo', recursive=True) >>> [root@shale foo]# ls -lZ mega -rw-r--r--. root root system_u:object_r:admin_home_t:s0 mega [root@shale foo]# ls -l /root/foo lrwxrwxrwx. 1 root root 9 Jan 12 11:11 /root/foo -> /root/bar [root@shale foo]# rpm -q libselinux-python libselinux-python-2.5-6.el7.x86_64 I'm failing to understand why this is failing. Does this happen with up-to-date libselinux/libselinux-python/python/etc? I can't get it to fail at all, even if I remove the target directory of the symlink: [root@shale foo]# cd /root [root@shale ~]# rm -rf bar [root@shale ~]# python Python 2.7.5 (default, Aug 2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import selinux >>> selinux.restorecon('/root/foo', recursive=True) >>> [root@shale ~]# ls -l /root/foo lrwxrwxrwx. 1 root root 9 Jan 12 11:11 /root/foo -> /root/bar [root@shale ~]# cd /root/foo -bash: cd: /root/foo: No such file or directory The only way I can get it to fail is if the file or a subdirectory was actually missing:
>>> selinux.restorecon('/root/foo/mega', recursive=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 84, in restorecon
mode = os.lstat(path)[stat.ST_MODE]
OSError: [Errno 2] No such file or directory: '/root/bar/mega'
So, it's likely things are actually not there.
Let's try something that actually exists but it doesn't have a SELinux label assigned by the policy. For example: # semanage fcontext -l | grep -i none /tmp/.* all files <<None>> /run/.*\.*pid all files <<None>> /mnt/[^/]*/.* all files <<None>> /proc/.* all files <<None>> /rhev/[^/]*/.* all files <<None>> /media/[^/]*/.* all files <<None>> /dev/pts(/.*)? all files <<None>> /usr/tmp/.* all files <<None>> /selinux/.* all files <<None>> /dev/shm/.* all files <<None>> /var/tmp/.* all files <<None>> /var/run/.*\.*pid all files <<None>> /var/lock/.* all files <<None>> /dev/mqueue(/.*)? all files <<None>> /lost\+found/.* all files <<None>> /root/\.debug(/.*)? all files <<None>> /var/run/shm/.* all files <<None>> /var/run/lock/.* all files <<None>> /dev/hugepages(/.*)? all files <<None>> /var/run/user/[^/]*/gvfs/.* all files <<None>> /var/run/media/.* all files <<None>> /sys/fs/cgroup/.* all files <<None>> /sys/fs/pstore/.* all files <<None>> /usr/lost\+found/.* all files <<None>> /tmp/lost\+found/.* all files <<None>> /var/lost\+found/.* all files <<None>> /var/spool/cron/[^/]* regular file <<None>> /boot/lost\+found/.* all files <<None>> /var/spool/fcron/.* all files <<None>> /var/log/cluster/.*\.*log all files <<None>> /sys/kernel/debug/.* all files <<None>> /var/run/initramfs(/.*)? all files <<None>> /var/log/lost\+found/.* all files <<None>> /var/tmp/lost\+found/.* all files <<None>> /var/named/chroot/proc(/.*)? all files <<None>> /var/lib/nfs/rpc_pipefs(/.*)? all files <<None>> /var/spool/cron/crontabs/.* regular file <<None>> /usr/lib/udev/devices/shm/.* all files <<None>> /var/log/audit/lost\+found/.* all files <<None>> /usr/lib/udev/devices/hugepages/.* all files <<None>> /proc directory <<None>> /selinux directory <<None>> /\.journal all files <<None>> /var/\.journal all files <<None>> /usr/\.journal all files <<None>> /tmp/\.journal all files <<None>> /boot/\.journal all files <<None>> # # ls -dZ /proc dr-xr-xr-x. root root system_u:object_r:proc_t:s0 /proc # python Python 2.7.5 (default, Aug 2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import selinux >>> selinux.restorecon('/proc', recursive=True) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 85, in restorecon status, context = matchpathcon(path, mode) OSError: [Errno 2] No such file or directory >>> Where's the chef rpm coming from? Ah, good point, Milos - /run doesn't have a context. My guess is that the chef RPM comes from https://www.chef.io/chef/get-chef/ This can be fixed by setting the chef 'directories' key by passing in cloud-config data source containing:
chef:
directories:
- /etc/chef
- /var/log/chef
- /var/lib/chef
- /var/cache/chef
- /var/backups/chef
- /run/chef
(or modifying cloud.cfg on the image to contain this prior to deploying).
We should probably produce a package that has the correct default path, but the solution in #13 ought to provide an immediate fix that does not require waiting for a new package. The root cause of this bug is probably https://bugzilla.redhat.com/show_bug.cgi?id=1406520 *** This bug has been marked as a duplicate of bug 1406520 *** I don't think this is a duplicate.
/run is labeled in the same way as /var/run:
# semanage fcontext -l
...
/run = /var/run
There are patterns is /var/run with'<<None>>' label but all of them are outside of /var/run/chef:
# semanage fcontext -l | grep '/var/run' | grep None
/var/run/.*\.*pid all files <<None>>
/var/run/shm/.* all files <<None>>
/var/run/lock/.* all files <<None>>
/var/run/user/[^/]*/gvfs/.* all files <<None>>
/var/run/media/.* all files <<None>>
/var/run/initramfs(/.*)? all files <<None>>
So unless cloudinit tries to relabel the whole '/var/run', it's not the same issue. And it was stated above that when '/var/run/chef' exists, selinux.restorecon('/var/run/chef') works as expected.
Could you please confirm that libselinux-2.5-7.el7 really fixes this issue?
(In reply to Petr Lautrbach from comment #17) > So unless cloudinit tries to relabel the whole '/var/run', it's not the same > issue. And it was stated above that when '/var/run/chef' exists, > selinux.restorecon('/var/run/chef') works as expected. Based on the following message in the reporters log, it's indeed a duplicate. Nov 16 15:52:28 vc2cint118486n cloud-init[1628]: [CLOUDINIT] util.py[DEBUG]: Restoring selinux mode for /run (recursive=True) Sorry for the noise. *** This bug has been marked as a duplicate of bug 1406520 *** |