Description of problem: After packstack all-in-one installation Neutron server fails with following traceback: 2014-05-16 14:28:36.975 20499 CRITICAL neutron [-] [Errno 13] Permission denied: '/tmp/keystone-signing-Ipk4ys' 2014-05-16 14:28:36.975 20499 TRACE neutron Traceback (most recent call last): 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/bin/neutron-server", line 10, in <module> 2014-05-16 14:28:36.975 20499 TRACE neutron sys.exit(main()) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/server/__init__.py", line 47, in main 2014-05-16 14:28:36.975 20499 TRACE neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 113, in serve_wsgi 2014-05-16 14:28:36.975 20499 TRACE neutron LOG.exception(_('Unrecoverable error: please check log ' 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 82, in __exit__ 2014-05-16 14:28:36.975 20499 TRACE neutron six.reraise(self.type_, self.value, self.tb) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 106, in serve_wsgi 2014-05-16 14:28:36.975 20499 TRACE neutron service.start() 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 75, in start 2014-05-16 14:28:36.975 20499 TRACE neutron self.wsgi_app = _run_wsgi(self.app_name) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 175, in _run_wsgi 2014-05-16 14:28:36.975 20499 TRACE neutron app = config.load_paste_app(app_name) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 170, in load_paste_app 2014-05-16 14:28:36.975 20499 TRACE neutron app = deploy.loadapp("config:%s" % config_path, name=app_name) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp 2014-05-16 14:28:36.975 20499 TRACE neutron return loadobj(APP, uri, name=name, **kw) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj 2014-05-16 14:28:36.975 20499 TRACE neutron return context.create() 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2014-05-16 14:28:36.975 20499 TRACE neutron return self.object_type.invoke(self) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke 2014-05-16 14:28:36.975 20499 TRACE neutron **context.local_conf) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call 2014-05-16 14:28:36.975 20499 TRACE neutron val = callable(*args, **kw) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory 2014-05-16 14:28:36.975 20499 TRACE neutron app = loader.get_app(app_name, global_conf=global_conf) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app 2014-05-16 14:28:36.975 20499 TRACE neutron name=name, global_conf=global_conf).create() 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2014-05-16 14:28:36.975 20499 TRACE neutron return self.object_type.invoke(self) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke 2014-05-16 14:28:36.975 20499 TRACE neutron **context.local_conf) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call 2014-05-16 14:28:36.975 20499 TRACE neutron val = callable(*args, **kw) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 72, in pipeline_factory 2014-05-16 14:28:36.975 20499 TRACE neutron app = filter(app) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py", line 1387, in auth_filter 2014-05-16 14:28:36.975 20499 TRACE neutron return AuthProtocol(app, conf) 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py", line 467, in __init__ 2014-05-16 14:28:36.975 20499 TRACE neutron self.signing_dirname = tempfile.mkdtemp(prefix='keystone-signing-') 2014-05-16 14:28:36.975 20499 TRACE neutron File "/usr/lib64/python2.7/tempfile.py", line 329, in mkdtemp 2014-05-16 14:28:36.975 20499 TRACE neutron _os.mkdir(file, 0700) 2014-05-16 14:28:36.975 20499 TRACE neutron OSError: [Errno 13] Permission denied: '/tmp/keystone-signing-Ipk4ys' 2014-05-16 14:28:36.975 20499 TRACE neutron Note that other services don't have problem creating tmp directories: [root@localhost ~]# ll /tmp/ total 4 drwx------. 2 nova nova 6 May 16 14:11 keystone-signing-1UMWN9 drwx------. 2 nova nova 6 May 16 14:11 keystone-signing-eB_7Of drwx------. 2 cinder cinder 6 May 16 14:10 keystone-signing-fgatlb drwx------. 2 nova nova 6 May 16 14:11 keystone-signing-MdZcVv drwx------. 2 cinder cinder 6 May 16 14:10 keystone-signing-wwWk3m Version-Release number of selected component (if applicable): [root@localhost ~]# rpm -qa openstack-neutron* openstack-neutron-2014.1-19.el7ost.noarch openstack-neutron-openvswitch-2014.1-19.el7ost.noarch
Are there any AVC violations in logs?
Ah right, selinux again. Yup there is one: [para@localhost ~]$ sudo ausearch -m avc | grep neutron type=SYSCALL msg=audit(1400496218.377:9082): arch=c000003e syscall=83 success=no exit=-13 a0=2fdbd40 a1=1c0 a2=7f33c074bf88 a3=7fff3be58d90 items=0 ppid=1 pid=22546 auid=4294967295 uid=994 gid=993 euid=994 suid=994 fsuid=994 egid=993 sgid=993 fsgid=993 tty=(none) ses=4294967295 comm="neutron-server" exe="/usr/bin/python2.7" subj=system_u:system_r:neutron_t:s0 key=(null) type=AVC msg=audit(1400496218.377:9082): avc: denied { create } for pid=22546 comm="neutron-server" name="keystone-signing-cWBO58" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir
Could you re-test it in permissive mode?
Actually what does show you rpm -q selinux-policy-targeted
In permissive mode Neutron server works OK. [para@localhost ~]$ rpm -q selinux-policy-targeted selinux-policy-targeted-3.12.1-153.el7.noarch
(In reply to Martin Magr from comment #6) > In permissive mode Neutron server works OK. > > [para@localhost ~]$ rpm -q selinux-policy-targeted > selinux-policy-targeted-3.12.1-153.el7.noarch Sure, I wanted to see AVC msgs.
Isn't 'ausearch' output enough?
Yes, it is. 'ausearch' in permissive mode after re-testing. And what does $ sesearch -T -s neutron_t -c dir |grep tmp_t
This should be in last night's puddle
Tested on selinux-policy-3.12.1-153.el7_0.10.noarch Jun 5 10:14:56 puma04 kernel: type=1400 audit(1401952496.719:11): avc: denied { create } for pid=22723 comm="neutron-server" name="keystone-signing-YDi1vz" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir
What does #sesearch -T -s neutron_t -c dir |grep tmp_t
*** Bug 1106296 has been marked as a duplicate of this bug. ***
bug 1106296 was a duplicate of this one - Giulio, were you using the newer package?
It appears that this was fixed by the updated package from Miroslav: https://bugzilla.redhat.com/show_bug.cgi?id=1099044#c3
This looks like upstream commit 30935facbaf4e1139088076e312070ea95293aa0 - I'll see if it's in the .10 package.
The .10 package contains the referenced fixes, yet, the problem persists. Apparently, rebooting cures the problem. type=AVC msg=audit(1400496218.377:9082): avc: denied { create } for pid=22546 comm="neutron-server" name="keystone-signing-cWBO58" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir Was caused by: Missing type enforcement (TE) allow rule. You can use audit2allow to generate a loadable module to allow this access. Audit2allow: require { type neutron_t; type tmp_t; class dir create; } #============= neutron_t ============== allow neutron_t tmp_t:dir create; This is interesting. I wonder if neutron is creating tmp files in the wrong place initially for some reason, so a reboot fixes it.
Prior to reboot, a user may get an error like this: 2014-06-10 21:59:43.602 22927 TRACE neutron.service OSError: [Errno 13] Permission denied: '/tmp/keystone-signing-igsY_2' After reboot, the temp files are in a different location, like this: /tmp/systemd-private-qsb5vM/tmp/keystone-signing-hFUcLA ... and things work.
Except, I can't reproduce the original behavior on a clean install with 7.0.z and the current latest available bits for RHEL OSP - [root@localhost tmp]# find /tmp -user neutron /tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC /tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC/signing_cert.pem /tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC/cacert.pem /tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC/revoked.pem [root@localhost tmp]# ls -lZ /tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC -rw-------. neutron neutron system_u:object_r:neutron_tmp_t:s0 cacert.pem -rw-------. neutron neutron system_u:object_r:neutron_tmp_t:s0 revoked.pem -rw-------. neutron neutron system_u:object_r:neutron_tmp_t:s0 signing_cert.pem [root@localhost tmp]# ls -ldZ /tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC drwx------. neutron neutron system_u:object_r:neutron_tmp_t:s0 /tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC
[root@localhost tmp]# ausearch -i -m avc <no matches> openstack-neutron-2014.1-34.el7ost.noarch selinux-policy-3.12.1-153.el7_0.10.noarch openstack-selinux-0.5.2-2.el7ost.noarch
Packstack in the collection you tested: ./packstack/puppet/templates/postscript.pp:exec { 'update-selinux-policy': ./packstack/puppet/templates/postscript.pp: command => "yum update -y selinux-policy-targeted" So, I think the problem is this: 1) Install RHEL 7.0 2) Use packstack to deploy At the end of installation, *after* tmpfiles are used by neutron (and presumably denied until then) - so, a reboot works. Now, with current packstack: ./packstack/puppet/templates/prescript.pp:# We don't have openstack-selinux package for Fedora ./packstack/puppet/templates/prescript.pp: package{ 'openstack-selinux': The openstack-selinux package depends on selinux-policy .10 release, meaning that it's installed before neutron is started. So, this should not occur any more. I'm trying to reproduce, however.
(In reply to Lon Hohberger from comment #33) > At the end of installation, *after* tmpfiles are used by neutron (and > presumably denied until then) - so, a reboot works. > That is - at the end of installation, selinux-policy is updated by packstack.
Reproduced. This is indeed the problem, and it will be resolved because openstack-selinux depends on the 7.0.z selinux-policy, which has the fixes Miroslav noted.
verified openstack-packstack-2014.1.1-0.26.dev1157.el7ost.noarch openstack-packstack-puppet-2014.1.1-0.26.dev1157.el7ost.noarch selinux-policy-3.12.1-153.el7_0.10.noarch openstack-selinux-0.5.2-2.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-0845.html