+++ This bug was initially created as a clone of Bug #1083566 +++ Description of problem: Instance launching fails on an AIO installation on RHEL 7. I've installed the Openstack without Neutron, it is working with Nova-network. When setting the SELinux to permissive it works well. The nova compute log show: 2014-04-02 16:12:46.405 10851 DEBUG nova.compute.utils [req-50d44af2-aeec-4f89-a557-fa94fc481eb8 4439a5fdc6c04d68aaf1115daa515234 3ab981eb8f4b465fb23ab505cc7bcf8b] [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] Timed out waiting for a reply to message ID 68db9e9cd6f449059d11d9da2c139fe9 notify_about_instance_usage /usr/lib/python2.7/site-packages/nova/compute/utils.py:335 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] Traceback (most recent call last): 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1066, in _build_instance 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] set_access_ip=set_access_ip) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360, in decorated_function 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] return function(self, context, *args, **kwargs) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1476, in _spawn 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] LOG.exception(_('Instance failed to spawn'), instance=instance) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__ 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] six.reraise(self.type_, self.value, self.tb) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1473, in _spawn 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] block_device_info) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2230, in spawn 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] admin_pass=admin_password) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2634, in _create_image 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] net = netutils.get_injected_network_template(network_info) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] if not (network_info and template): 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 379, in __len__ 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] return self._sync_wrapper(fn, *args, **kwargs) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 366, in _sync_wrapper 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] self.wait() 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 398, in wait 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] self[:] = self._gt.wait() 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] return self._exit_event.wait() 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] return hubs.get_hub().switch() 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] return self.greenlet.switch() 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] result = function(*args, **kwargs) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1254, in _allocate_network_async 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] dhcp_options=dhcp_options) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 94, in wrapped 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] return func(self, context, *args, **kwargs) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 48, in wrapper 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] res = f(self, context, *args, **kwargs) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 302, in allocate_for_instance 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] nw_info = self.network_rpcapi.allocate_for_instance(context, **args) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 170, in allocate_for_instance 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] macs=jsonutils.to_primitive(macs)) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] wait_for_reply=True, timeout=timeout) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] timeout=timeout) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 409, in send 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] return self._send(target, ctxt, message, wait_for_reply, timeout) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 400, in _send 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] result = self._waiter.wait(msg_id, timeout) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 280, in wait 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] reply, ending, trylock = self._poll_queue(msg_id, timeout) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 220, in _poll_queue 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] message = self.waiters.get(msg_id, timeout) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 126, in get 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] 'to message ID %s' % msg_id) 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] MessagingTimeout: Timed out waiting for a reply to message ID 68db9e9cd6f449059d11d9da2c139fe9 2014-04-02 16:12:46.405 10851 TRACE nova.compute.utils [instance: 67082855-3395-4cc0-949b-3e2a693b6cf5] The nova network log show: 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup Traceback (most recent call last): 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 117, in wait 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup x.wait() 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 49, in wait 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup return self.thread.wait() 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup return self._exit_event.wait() 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch() 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup return self.greenlet.switch() 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs) 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 480, in run_service 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup service.start() 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/service.py", line 180, in start 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup self.manager.init_host() 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/manager.py", line 1649, in init_host 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup self.l3driver.initialize(fixed_range=False, networks=networks) 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/l3.py", line 88, in initialize 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup self.initialize_network(network['cidr']) 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/l3.py", line 99, in initialize_network 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup linux_net.init_host(cidr) 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 679, in init_host 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup add_snat_rule(ip_range) 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 671, in add_snat_rule 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup iptables_manager.apply() 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 428, in apply 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup self._apply() 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 249, in inner 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup return f(*args, **kwargs) 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 448, in _apply 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup attempts=5) 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 1206, in _execute 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup return utils.execute(*cmd, **kwargs) 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/utils.py", line 164, in execute 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup return processutils.execute(*cmd, **kwargs) 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 193, in execute 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup cmd=' '.join(cmd)) 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup ProcessExecutionError: Unexpected error while running command. 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup Exit code: 1 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup Stdout: '' 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup Stderr: 'sudo: unknown uid 162: who are you?\n' 2014-04-02 15:47:24.341 10905 TRACE nova.openstack.common.threadgroup Version-Release number of selected component (if applicable): openstack-puppet-modules-2014.1-5.3.el7.noarch openstack-nova-conductor-2014.1-0.13.b3.el7.noarch openstack-swift-object-1.12.0-1.el7.noarch openstack-nova-common-2014.1-0.13.b3.el7.noarch openstack-packstack-2014.1.1-0.7.dev1018.el7.noarch openstack-nova-cert-2014.1-0.13.b3.el7.noarch python-django-openstack-auth-1.1.4-1.el7.noarch openstack-swift-1.12.0-1.el7.noarch openstack-glance-2014.1-0.4.b3.el7.noarch openstack-packstack-puppet-2014.1.1-0.7.dev1018.el7.noarch openstack-nova-compute-2014.1-0.13.b3.el7.noarch openstack-nova-novncproxy-2014.1-0.13.b3.el7.noarch openstack-utils-2013.2-3.p1.el7.noarch openstack-dashboard-2014.1-0.5.b3.el7.noarch openstack-swift-account-1.12.0-1.el7.noarch openstack-swift-proxy-1.12.0-1.el7.noarch openstack-nova-network-2014.1-0.13.b3.el7.noarch openstack-keystone-2014.1-0.4.b3.el7.noarch openstack-nova-api-2014.1-0.13.b3.el7.noarch openstack-swift-container-1.12.0-1.el7.noarch openstack-nova-scheduler-2014.1-0.13.b3.el7.noarch openstack-nova-console-2014.1-0.13.b3.el7.noarch openstack-swift-plugin-swift3-1.7-3.el7.noarch openstack-cinder-2014.1-0.6.b3.el7.noarch How reproducible: 100% Steps to Reproduce: 1. Install openstack AIO with packstack on RHEL 7. 2. Upload the CirrOS to the Glance. 3. Launch an instance with that image. Actual results: The launch fail and the instance's status is error. Expected results: The launch should succeed and the instance should be active. Additional info: --- Additional comment from Ryan Hallisey on 2014-04-21 15:06:08 EDT --- Can you attach /var/log/audit/audit.log please? --- Additional comment from Yogev Rabl on 2014-04-22 04:01:47 EDT --- --- Additional comment from Yogev Rabl on 2014-04-22 04:33:51 EDT --- --- Additional comment from Tzach Shefi on 2014-04-22 06:52:47 EDT --- Ran into another aspect of selinux blocks Nova's service problem. RDO on RHEL7, default selinux 1. On instance boot status-> error power->no state, other than not being able to boot instances, they can't be terminated either. Nova delete <> ,doesn't delete instance, stuck in "deleting" Nova reset-state --active <> and Nova delete <> ,also stuck in deleting state When setenforce 0, above steps successfully terminate/delete stuck instances. --- Additional comment from Ryan Hallisey on 2014-04-23 15:57:12 EDT --- The audit log during launch doesn't have any avc's, but the audit log for running openstack-service restart does. This is a little bizarre considering that you can boot in permissive and not in enforcing. I have the policy changes to fix openstack-service restart, but first try booting an instance again and grab the audit log to see if it's different. I would except you would see some avcs for it to fail in enforcing. I reproduced booting an instance with RDO on rhel7 and got some a few avcs which I will attach in the audit log. $ rpm -qa selinux-policy selinux-policy-3.12.1-153.el7.noarch --- Additional comment from Ryan Hallisey on 2014-04-23 16:03:08 EDT --- Search for 'nova' and you will see some avcs --- Additional comment from Yogev Rabl on 2014-05-01 05:14:00 EDT --- When restarting the openstack services I can avcs: # grep nova audit.log | grep avc output: type=AVC msg=audit(1398935297.980:93): avc: denied { signal } for pid=1133 comm="nova-api" scontext=system_u:system_r:nova_api_t:s0 tcontext=system_u:system_r:nova_api_t:s0 tclass=process type=AVC msg=audit(1398935298.855:112): avc: denied { read } for pid=4697 comm="sh" name="passwd" dev="sda5" ino=24118430 scontext=system_u:system_r:nova_network_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file type=AVC msg=audit(1398935298.861:113): avc: denied { read } for pid=4699 comm="sh" name="passwd" dev="sda5" ino=24118430 scontext=system_u:system_r:nova_network_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file type=AVC msg=audit(1398935299.511:116): avc: denied { read } for pid=4737 comm="sh" name="passwd" dev="sda5" ino=24118430 scontext=system_u:system_r:nova_network_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file type=AVC msg=audit(1398935299.572:117): avc: denied { read } for pid=4740 comm="sh" name="passwd" dev="sda5" ino=24118430 scontext=system_u:system_r:nova_scheduler_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file type=AVC msg=audit(1398935299.583:118): avc: denied { read } for pid=4747 comm="sh" name="passwd" dev="sda5" ino=24118430 scontext=system_u:system_r:nova_scheduler_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file type=AVC msg=audit(1398935300.030:123): avc: denied { read } for pid=4779 comm="sh" name="passwd" dev="sda5" ino=24118430 scontext=system_u:system_r:nova_scheduler_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file type=AVC msg=audit(1398935301.511:149): avc: denied { read } for pid=4936 comm="sudo" name="passwd" dev="sda5" ino=24118430 scontext=system_u:system_r:nova_network_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file type=AVC msg=audit(1398935301.511:150): avc: denied { read } for pid=4936 comm="sudo" name="utmp" dev="tmpfs" ino=13785 scontext=system_u:system_r:nova_network_t:s0 tcontext=system_u:object_r:initrc_var_run_t:s0 tclass=file --- Additional comment from Ryan Hallisey on 2014-06-04 17:27:10 EDT --- Using the updated policy, selinux-policy-3.12.1-153.el7_0.10, only yeilds one avc. #============= nova_network_t ============== allow nova_network_t initrc_var_run_t:file read; This can be shipped in selinux policy for RHEL 6.6 & 7. --- Additional comment from Miroslav Grepl on 2014-06-05 10:21:44 EDT --- I think it's too late for our rhel-7.0.z errata which we have for non OpenStack issue add these. Milos, how does it look? Do you agree with me? --- Additional comment from Milos Malik on 2014-06-05 11:31:44 EDT --- The 0day selinux-policy errata for RHEL-7.0 must go to REL_PREP tomorrow (June 6). If you do a new build of selinux-policy during today night I can test the differences in the morning and the errata gets pushed as planned. --- Additional comment from Miroslav Grepl on 2014-06-06 05:23:11 EDT --- Yes, so too late. We will need to have a new bug for these OpenStack issues.
Was able to reproduce this under packstack with rhel osp 5. created vm from rhel7 qcow2. Applied all updates yum -y update Setup rhos-release rpm -ivh http://team.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm; rhos-release 5 Installed packstack yum -y install openstack-packstack Did packstack all in one install packstack --allinone --os-neutron-install=n --os-ceilometer-install=n --os-horizon-install=n Install completed successfully: Applying Puppet manifests [ DONE ] Finalizing [ DONE ] **** Installation completed successfully ****** Additional information: * A new answerfile was created in: /root/packstack-answers-20140610-203026.txt * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components. * File /root/keystonerc_admin has been created on OpenStack client host 192.168.122.68. To use the command line tools you need to source the file. * To use Nagios, browse to http://192.168.122.68/nagios username: nagiosadmin, password: 669a8cb15ffe44d1 * The installation log file is available at: /var/tmp/packstack/20140610-203026-03r_7y/openstack-setup.log * The generated manifests are available at: /var/tmp/packstack/20140610-203026-03r_7y/manifests SELinux enforcing: [root@packstack ~]# getenforce Enforcing Same traceback/error from openstack-nova-network log: [root@packstack ~]# journalctl --full -u openstack-nova-network -- Logs begin at Tue 2014-06-10 20:28:13 EDT, end at Tue 2014-06-10 20:37:37 EDT. -- Jun 10 20:34:30 packstack.localdomain systemd[1]: Starting OpenStack Nova Network Server... Jun 10 20:34:30 packstack.localdomain systemd[1]: Started OpenStack Nova Network Server. Jun 10 20:34:31 packstack.localdomain nova-network[10630]: Traceback (most recent call last): Jun 10 20:34:31 packstack.localdomain nova-network[10630]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in fire_timers Jun 10 20:34:31 packstack.localdomain nova-network[10630]: timer() Jun 10 20:34:31 packstack.localdomain nova-network[10630]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 56, in __call__ Jun 10 20:34:31 packstack.localdomain nova-network[10630]: cb(*args, **kw) Jun 10 20:34:31 packstack.localdomain nova-network[10630]: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main Jun 10 20:34:31 packstack.localdomain nova-network[10630]: result = function(*args, **kwargs) Jun 10 20:34:31 packstack.localdomain nova-network[10630]: File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 483, in run_service Jun 10 20:34:31 packstack.localdomain nova-network[10630]: service.start() Jun 10 20:34:31 packstack.localdomain nova-network[10630]: File "/usr/lib/python2.7/site-packages/nova/service.py", line 163, in start Jun 10 20:34:31 packstack.localdomain nova-network[10630]: self.manager.init_host() Jun 10 20:34:31 packstack.localdomain nova-network[10630]: File "/usr/lib/python2.7/site-packages/nova/network/manager.py", line 1651, in init_host Jun 10 20:34:31 packstack.localdomain nova-network[10630]: self.l3driver.initialize(fixed_range=False, networks=networks) Jun 10 20:34:31 packstack.localdomain nova-network[10630]: File "/usr/lib/python2.7/site-packages/nova/network/l3.py", line 91, in initialize Jun 10 20:34:31 packstack.localdomain nova-network[10630]: linux_net.ensure_metadata_ip() Jun 10 20:34:31 packstack.localdomain nova-network[10630]: File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 737, in ensure_metadata_ip Jun 10 20:34:31 packstack.localdomain nova-network[10630]: run_as_root=True, check_exit_code=[0, 2, 254]) Jun 10 20:34:31 packstack.localdomain nova-network[10630]: File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 1205, in _execute Jun 10 20:34:31 packstack.localdomain nova-network[10630]: return utils.execute(*cmd, **kwargs) Jun 10 20:34:31 packstack.localdomain nova-network[10630]: File "/usr/lib/python2.7/site-packages/nova/utils.py", line 164, in execute Jun 10 20:34:31 packstack.localdomain nova-network[10630]: return processutils.execute(*cmd, **kwargs) Jun 10 20:34:31 packstack.localdomain nova-network[10630]: File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 193, in execute Jun 10 20:34:31 packstack.localdomain nova-network[10630]: cmd=' '.join(cmd)) Jun 10 20:34:31 packstack.localdomain nova-network[10630]: ProcessExecutionError: Unexpected error while running command. Jun 10 20:34:31 packstack.localdomain nova-network[10630]: Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ip addr add 169.254.169.254/32 scope link dev lo Jun 10 20:34:31 packstack.localdomain nova-network[10630]: Exit code: 1 Jun 10 20:34:31 packstack.localdomain nova-network[10630]: Stdout: '' Jun 10 20:34:31 packstack.localdomain nova-network[10630]: Stderr: 'sudo: unknown uid 996: who are you?\n' AVC from /var/log/audit/audit.log: type=AVC msg=audit(1402446871.526:2907): avc: denied { read } for pid=10682 comm="sudo" name="utmp" dev="tmpfs" ino=12245 scontext=system_u:system_r:nova_network_t:s0 tcontext=system_u:object_r:initrc_var_run_t:s0 tclass=file Current packages: [root@packstack ~]# rpm -q selinux-policy selinux-policy-targeted openstack-nova-network selinux-policy-3.12.1-153.el7_0.10.noarch selinux-policy-targeted-3.12.1-153.el7_0.10.noarch openstack-nova-network-2014.1-3.el7ost.noarch
Momentarily switching disabling selinux allows the openstack-nova-network service to start successfully. [root@packstack ~]# systemctl status openstack-nova-network openstack-nova-network.service - OpenStack Nova Network Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-network.service; enabled) Active: inactive (dead) since Tue 2014-06-10 20:43:34 EDT; 1s ago Process: 16145 ExecStart=/usr/bin/nova-network (code=exited, status=0/SUCCESS) Main PID: 16145 (code=exited, status=0/SUCCESS) CGroup: /system.slice/openstack-nova-network.service Jun 10 20:43:34 packstack.localdomain nova-network[16145]: return utils.execute(*cmd, **kwargs) Jun 10 20:43:34 packstack.localdomain nova-network[16145]: File "/usr/lib/python2.7/site-packages/nova/utils.py", line 164, in execute Jun 10 20:43:34 packstack.localdomain nova-network[16145]: return processutils.execute(*cmd, **kwargs) Jun 10 20:43:34 packstack.localdomain nova-network[16145]: File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 193, in execute Jun 10 20:43:34 packstack.localdomain nova-network[16145]: cmd=' '.join(cmd)) Jun 10 20:43:34 packstack.localdomain nova-network[16145]: ProcessExecutionError: Unexpected error while running command. Jun 10 20:43:34 packstack.localdomain nova-network[16145]: Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ip addr add 169.254.169.254/32 scope link dev lo Jun 10 20:43:34 packstack.localdomain nova-network[16145]: Exit code: 1 Jun 10 20:43:34 packstack.localdomain nova-network[16145]: Stdout: '' Jun 10 20:43:34 packstack.localdomain nova-network[16145]: Stderr: 'sudo: unknown uid 996: who are you?\n' [root@packstack ~]# [root@packstack ~]# setenforce 0 [root@packstack ~]# [root@packstack ~]# systemctl restart openstack-nova-network [root@packstack ~]# [root@packstack ~]# systemctl status openstack-nova-network openstack-nova-network.service - OpenStack Nova Network Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-network.service; enabled) Active: active (running) since Tue 2014-06-10 20:43:42 EDT; 1s ago Main PID: 16165 (nova-network) CGroup: /system.slice/openstack-nova-network.service └─16165 /usr/bin/python /usr/bin/nova-network Jun 10 20:43:42 packstack.localdomain systemd[1]: Starting OpenStack Nova Network Server... Jun 10 20:43:42 packstack.localdomain systemd[1]: Started OpenStack Nova Network Server. Jun 10 20:43:43 packstack.localdomain sudo[16172]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf ip addr add 169...nk dev lo Jun 10 20:43:43 packstack.localdomain sudo[16175]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c Jun 10 20:43:43 packstack.localdomain sudo[16178]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c Hint: Some lines were ellipsized, use -l to show in full.
audit.log: https://bugzilla.redhat.com/attachment.cgi?id=888379
I reproduced and created a local policy using audit2allow: module local 1.0; require { type initrc_var_run_t; type passwd_file_t; type nova_network_t; class file { read getattr open }; } #============= nova_network_t ============== allow nova_network_t initrc_var_run_t:file read; allow nova_network_t passwd_file_t:file { read getattr open }; After installing the policy I was able to run nova-network with Enforcing.
commit 67d09a639b7fc76379d3002969416e000d0676ed Author: Miroslav Grepl <mgrepl> Date: Wed Jun 11 10:47:40 2014 +0200 Allow nova domains to read passwd/utmp files
The AVCs / policy changes are covered by the other nova bug, except the init_read_utmp(nova_domain), which I didn't hit (different use case) ?
*** This bug has been marked as a duplicate of bug 1095869 ***