Hide Forgot
Description of problem:Specifically, cloudinit fails to install chef (see example http://cloudinit.readthedocs.io/en/latest/topics/examples.html#install-and-run-chef-recipes ) on a RHEL 7 server with selinux enabled in permissive mode. Nov 16 15:52:28 vc2cint118486n cloud-init[1628]: [CLOUDINIT] util.py[DEBUG]: /var/run/chef Nov 16 15:52:28 vc2cint118486n cloud-init[1628]: [CLOUDINIT] util.py[DEBUG]: Restoring selinux mode for /run (recursive=True) Nov 16 15:52:28 vc2cint118486n cloud-init[1628]: [CLOUDINIT] util.py[WARNING]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.pyc'>) failed Nov 16 15:52:28 vc2cint118486n cloud-init[1628]: 2016-11-16 15:52:28,810 - util.py[WARNING]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.pyc'>) failed Nov 16 15:52:28 vc2cint118486n cloud-init[1628]: [CLOUDINIT] util.py[DEBUG]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.pyc'>) failed Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cloudinit/stages.py", line 658, in _run_modules cc.run(run_name, mod.handle, func_args, freq=freq) File "/usr/lib/python2.7/site-packages/cloudinit/cloud.py", line 63, in run return self._runners.run(name, functor, args, freq, clear_on_fail) File "/usr/lib/python2.7/site-packages/cloudinit/helpers.py", line 197, in run results = functor(*args) File "/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.py", line 53, in handle util.ensure_dir(d) File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1290, in ensure_dir os.makedirs(path) File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 165, in __exit__ self.selinux.restorecon(path, recursive=self.recursive) File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 95, in restorecon for fname in fnames]), None) File "/usr/lib64/python2.7/posixpath.py", line 238, in walk func(arg, top, names) File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 95, in <lambda> for fname in fnames]), None) File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 85, in restorecon status, context = matchpathcon(path, mode) OSError: [Errno 2] No such file or directory The bug is selinux.restorecon doesn’t follow symlinks and /var/run is a symlink, so when it calls selinux.restorecon(‘/var/run/chef’) it can’t get to that path Version-Release number of selected component (if applicable): libselinux-2.2.2-6.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. 2. 3. Actual results: fails to install with selinux enabled Expected results: installed with selinux enabled. Additional info: Seems to be related to this bug: https://bugs.launchpad.net/cloud-init/+bug/1385860
RHEL-7.1 ======== # rpm -q libselinux libselinux-2.2.2-6.el7.x86_64 # python Python 2.7.5 (default, Feb 11 2014, 07:46:25) [GCC 4.8.2 20140120 (Red Hat 4.8.2-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import selinux >>> selinux.restorecon('/var/run/chef') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 84, in restorecon mode = os.lstat(path)[stat.ST_MODE] OSError: [Errno 2] No such file or directory: '/run/chef' >>> # RHEL-7.3 ======== # rpm -q libselinux libselinux-2.5-6.el7.x86_64 # python Python 2.7.5 (default, Aug 2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import selinux >>> selinux.restorecon('/var/run/chef') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 84, in restorecon mode = os.lstat(path)[stat.ST_MODE] OSError: [Errno 2] No such file or directory: '/run/chef' >>> #
> selinux.restorecon(‘/var/run/chef’) ^^^^^^^^^ > OSError: [Errno 2] No such file or directory: '/run/chef' ^^^^^ Is this a problem is libselinux? The /var/run symlink is apparently followed and it looks like /run/chef doesn't exist. When it exists, restorecon works as expected: >>> selinux.restorecon('/var/run/chef') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 84, in restorecon mode = os.lstat(path)[stat.ST_MODE] OSError: [Errno 2] No such file or directory: '/run/chef' >>> os.mkdir("/run/chef") >>> selinux.restorecon('/var/run/chef') >>>
Here is the output of ls -l /var/run along with some more background. Can you try to reproduce the error using the same steps. It looks like other users have reported the same issue at https://bugs.launchpad.net/cloud-init/+bug/1385860 . [root@hostname ~]# ls -l /var/run lrwxrwxrwx. 1 root root 6 Aug 1 14:25 /var/run -> ../run ======== Background ========== We are trying to create an instance of Red Hat Enterprise Linux Server release 7.2 (Maipo) on Red Hat Openstack Platform 8. The image has selinux enabled in permissive mode. When we send a request to create a new instance, we provide a cloud-init file which includes a section to install and run chef-client using the "omnibus" install_package as described at http://cloudinit.readthedocs.io/en/latest/topics/examples.html#install-and-run-chef-recipes . Here’s an example of the chef section, with sensitive data replaced with example data. Example chef config: #cloud-config --- chef: environment: example-environment exec: true force_install: false install_type: omnibus log_level: info node_name: ‘example12345’ omnibus_url: http://c2c-repo1-d.fmr.com/chef/omnibus/install.sh server_url: https://c2cchefbosbe.fmr.com/organizations/c2c-int-dev validation_key: '**key**' ====== Problem ================= During instance creation cloud-init fails on the chef section when cloud-init sets up the required directories. Here’s an example stack trace. We can reproduce this on demand using the steps below. Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: In ensure_dir path: /var/run/chef Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: PATH: /run Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: Restoring selinux mode for /run (recursive=True) Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: 2016-11-29 21:04:37,208 - util.py[WARNING]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[WARNING]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.pyc'> Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.pyc'>) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cloudinit/stages.py", line 658, in _run_modules cc.run(run_name, mod.handle, func_args, freq=freq) File "/usr/lib/python2.7/site-packages/cloudinit/cloud.py", line 63, in run return self._runners.run(name, functor, args, freq, clear_on_fail) File "/usr/lib/python2.7/site-packages/cloudinit/helpers.py", line 197, in run results = functor(*args) File "/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.py", line 53, in handle util.ensure_dir(d) File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1291, in ensure_dir os.makedirs(path) File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 166, in __exit__ self.selinux.restorecon(path, recursive=self.recursive) File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 95, in restorecon for fname in fnames]), None) File "/usr/lib64/python2.7/posixpath.py", line 238, in walk func(arg, top, names) File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 95, in <lambda> for fname in fnames]), None) File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 85, in restorecon status, context = matchpathcon(path, mode) OSError: [Errno 2] No such file or directory ===== Steps to Reproduce ========== Create an instance of Red Hat Enterprise Linux Server release 7.2 (Maipo) with selinux enabled in permissive mode on RHOSP 8 with a cloud-init file that installs chef using install
it should be creating "/var/run/chef" directory as part of it's operation, so that not existing shouldn't be the issue here. In cloud-init's cc_chef.py: <snip> CHEF_DIRS = [ '/etc/chef', '/var/log/chef', '/var/lib/chef', '/var/cache/chef', '/var/backups/chef', '/var/run/chef', ] <snip> # Ensure the chef directories we use exist for d in CHEF_DIRS: util.ensure_dir(d) <snip> cloudinit/util.py: <snip> def ensure_dir(path, mode=None): if not os.path.isdir(path): # Make the dir and adjust the mode with SeLinuxGuard(os.path.dirname(path), recursive=True): os.makedirs(path) chmod(path, mode) else: # Just adjust the mode chmod(path, mode) <snip> class SeLinuxGuard(object): def __init__(self, path, recursive=False): # Late import since it might not always # be possible to use this try: self.selinux = importer.import_module('selinux') except ImportError: self.selinux = None self.path = path self.recursive = recursive def __enter__(self): if self.selinux and self.selinux.is_selinux_enabled(): return True else: return False def __exit__(self, excp_type, excp_value, excp_traceback): if not self.selinux or not self.selinux.is_selinux_enabled(): return if not os.path.lexists(self.path): return path = os.path.realpath(self.path) # path should be a string, not unicode path = str(path) try: stats = os.lstat(path) self.selinux.matchpathcon(path, stats[stat.ST_MODE]) except OSError: return LOG.debug("Restoring selinux mode for %s (recursive=%s)", path, self.recursive) self.selinux.restorecon(path, recursive=self.recursive) <snip> So, we're calling 'restorecon' from the selinux module: site-packages/selinux/__init__.py <snip> def restorecon(path, recursive=False): """ Restore SELinux context on a given path """ try: mode = os.lstat(path)[stat.ST_MODE] status, context = matchpathcon(path, mode) except OSError: path = os.path.realpath(os.path.expanduser(path)) mode = os.lstat(path)[stat.ST_MODE] status, context = matchpathcon(path, mode) ... <snip> That's where we're failing per the error log: <error> Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: In ensure_dir path: /var/run/chef Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: PATH: /run Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: Restoring selinux mode for /run (recursive=True) Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: 2016-11-29 21:04:37,208 - util.py[WARNING]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[WARNING]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.pyc'> Nov 29 21:04:37 vc2cint119840n cloud-init[1589]: [CLOUDINIT] util.py[DEBUG]: Running chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.pyc'>) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cloudinit/stages.py", line 658, in _run_modules cc.run(run_name, mod.handle, func_args, freq=freq) File "/usr/lib/python2.7/site-packages/cloudinit/cloud.py", line 63, in run return self._runners.run(name, functor, args, freq, clear_on_fail) File "/usr/lib/python2.7/site-packages/cloudinit/helpers.py", line 197, in run results = functor(*args) File "/usr/lib/python2.7/site-packages/cloudinit/config/cc_chef.py", line 53, in handle util.ensure_dir(d) File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1291, in ensure_dir os.makedirs(path) File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 166, in __exit__ self.selinux.restorecon(path, recursive=self.recursive) File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 95, in restorecon for fname in fnames]), None) File "/usr/lib64/python2.7/posixpath.py", line 238, in walk func(arg, top, names) File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 95, in <lambda> for fname in fnames]), None) File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 85, in restorecon status, context = matchpathcon(path, mode) OSError: [Errno 2] No such file or directory </snip> Again, it's not immediately clear why 'matchpathcon' would be having issue here. There were a few fixes earlier in RHEL's life to allow it to handle relative paths and the like, but I don't see anything outstanding that's interesting. I'm still not quite clear what it's trying that's not available to it at runtime, so we need to do a bit more digging here.
[root@shale foo]# chcon system_u:object_r:var_run_t mega [root@shale foo]# ls -lZ mega -rw-r--r--. root root system_u:object_r:var_run_t mega [root@shale foo]# python Python 2.7.5 (default, Aug 2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import selinux >>> selinux.restorecon('/root/foo', recursive=True) >>> [root@shale foo]# ls -lZ mega -rw-r--r--. root root system_u:object_r:admin_home_t:s0 mega [root@shale foo]# ls -l /root/foo lrwxrwxrwx. 1 root root 9 Jan 12 11:11 /root/foo -> /root/bar [root@shale foo]# rpm -q libselinux-python libselinux-python-2.5-6.el7.x86_64 I'm failing to understand why this is failing. Does this happen with up-to-date libselinux/libselinux-python/python/etc? I can't get it to fail at all, even if I remove the target directory of the symlink: [root@shale foo]# cd /root [root@shale ~]# rm -rf bar [root@shale ~]# python Python 2.7.5 (default, Aug 2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import selinux >>> selinux.restorecon('/root/foo', recursive=True) >>> [root@shale ~]# ls -l /root/foo lrwxrwxrwx. 1 root root 9 Jan 12 11:11 /root/foo -> /root/bar [root@shale ~]# cd /root/foo -bash: cd: /root/foo: No such file or directory
The only way I can get it to fail is if the file or a subdirectory was actually missing: >>> selinux.restorecon('/root/foo/mega', recursive=True) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 84, in restorecon mode = os.lstat(path)[stat.ST_MODE] OSError: [Errno 2] No such file or directory: '/root/bar/mega' So, it's likely things are actually not there.
Let's try something that actually exists but it doesn't have a SELinux label assigned by the policy. For example: # semanage fcontext -l | grep -i none /tmp/.* all files <<None>> /run/.*\.*pid all files <<None>> /mnt/[^/]*/.* all files <<None>> /proc/.* all files <<None>> /rhev/[^/]*/.* all files <<None>> /media/[^/]*/.* all files <<None>> /dev/pts(/.*)? all files <<None>> /usr/tmp/.* all files <<None>> /selinux/.* all files <<None>> /dev/shm/.* all files <<None>> /var/tmp/.* all files <<None>> /var/run/.*\.*pid all files <<None>> /var/lock/.* all files <<None>> /dev/mqueue(/.*)? all files <<None>> /lost\+found/.* all files <<None>> /root/\.debug(/.*)? all files <<None>> /var/run/shm/.* all files <<None>> /var/run/lock/.* all files <<None>> /dev/hugepages(/.*)? all files <<None>> /var/run/user/[^/]*/gvfs/.* all files <<None>> /var/run/media/.* all files <<None>> /sys/fs/cgroup/.* all files <<None>> /sys/fs/pstore/.* all files <<None>> /usr/lost\+found/.* all files <<None>> /tmp/lost\+found/.* all files <<None>> /var/lost\+found/.* all files <<None>> /var/spool/cron/[^/]* regular file <<None>> /boot/lost\+found/.* all files <<None>> /var/spool/fcron/.* all files <<None>> /var/log/cluster/.*\.*log all files <<None>> /sys/kernel/debug/.* all files <<None>> /var/run/initramfs(/.*)? all files <<None>> /var/log/lost\+found/.* all files <<None>> /var/tmp/lost\+found/.* all files <<None>> /var/named/chroot/proc(/.*)? all files <<None>> /var/lib/nfs/rpc_pipefs(/.*)? all files <<None>> /var/spool/cron/crontabs/.* regular file <<None>> /usr/lib/udev/devices/shm/.* all files <<None>> /var/log/audit/lost\+found/.* all files <<None>> /usr/lib/udev/devices/hugepages/.* all files <<None>> /proc directory <<None>> /selinux directory <<None>> /\.journal all files <<None>> /var/\.journal all files <<None>> /usr/\.journal all files <<None>> /tmp/\.journal all files <<None>> /boot/\.journal all files <<None>> # # ls -dZ /proc dr-xr-xr-x. root root system_u:object_r:proc_t:s0 /proc # python Python 2.7.5 (default, Aug 2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import selinux >>> selinux.restorecon('/proc', recursive=True) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/site-packages/selinux/__init__.py", line 85, in restorecon status, context = matchpathcon(path, mode) OSError: [Errno 2] No such file or directory >>>
Where's the chef rpm coming from?
Ah, good point, Milos - /run doesn't have a context.
My guess is that the chef RPM comes from https://www.chef.io/chef/get-chef/
This can be fixed by setting the chef 'directories' key by passing in cloud-config data source containing: chef: directories: - /etc/chef - /var/log/chef - /var/lib/chef - /var/cache/chef - /var/backups/chef - /run/chef (or modifying cloud.cfg on the image to contain this prior to deploying).
We should probably produce a package that has the correct default path, but the solution in #13 ought to provide an immediate fix that does not require waiting for a new package.
The root cause of this bug is probably https://bugzilla.redhat.com/show_bug.cgi?id=1406520
*** This bug has been marked as a duplicate of bug 1406520 ***
I don't think this is a duplicate. /run is labeled in the same way as /var/run: # semanage fcontext -l ... /run = /var/run There are patterns is /var/run with'<<None>>' label but all of them are outside of /var/run/chef: # semanage fcontext -l | grep '/var/run' | grep None /var/run/.*\.*pid all files <<None>> /var/run/shm/.* all files <<None>> /var/run/lock/.* all files <<None>> /var/run/user/[^/]*/gvfs/.* all files <<None>> /var/run/media/.* all files <<None>> /var/run/initramfs(/.*)? all files <<None>> So unless cloudinit tries to relabel the whole '/var/run', it's not the same issue. And it was stated above that when '/var/run/chef' exists, selinux.restorecon('/var/run/chef') works as expected. Could you please confirm that libselinux-2.5-7.el7 really fixes this issue?
(In reply to Petr Lautrbach from comment #17) > So unless cloudinit tries to relabel the whole '/var/run', it's not the same > issue. And it was stated above that when '/var/run/chef' exists, > selinux.restorecon('/var/run/chef') works as expected. Based on the following message in the reporters log, it's indeed a duplicate. Nov 16 15:52:28 vc2cint118486n cloud-init[1628]: [CLOUDINIT] util.py[DEBUG]: Restoring selinux mode for /run (recursive=True) Sorry for the noise. *** This bug has been marked as a duplicate of bug 1406520 ***