Bug 1098524 - Neutron is unable to create directory in /tmp
Summary: Neutron is unable to create directory in /tmp
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux
Version: 5.0 (RHEL 7)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 5.0 (RHEL 7)
Assignee: Lon Hohberger
QA Contact: Nir Magnezi
URL:
Whiteboard:
: 1106296 (view as bug list)
Depends On:
Blocks: 1099040 1099042 1099044
TreeView+ depends on / blocked
 
Reported: 2014-05-16 12:43 UTC by Martin Magr
Modified: 2016-04-26 18:35 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1099040 1099042 1099044 (view as bug list)
Environment:
Last Closed: 2014-07-08 15:13:27 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2014:0845 0 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform Enhancement - Runtime Components 2014-07-08 19:11:27 UTC

Description Martin Magr 2014-05-16 12:43:13 UTC
Description of problem:
After packstack all-in-one installation Neutron server fails with following traceback:
2014-05-16 14:28:36.975 20499 CRITICAL neutron [-] [Errno 13] Permission denied: '/tmp/keystone-signing-Ipk4ys'
2014-05-16 14:28:36.975 20499 TRACE neutron Traceback (most recent call last):
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/bin/neutron-server", line 10, in <module>
2014-05-16 14:28:36.975 20499 TRACE neutron     sys.exit(main())
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/server/__init__.py", line 47, in main
2014-05-16 14:28:36.975 20499 TRACE neutron     neutron_api = service.serve_wsgi(service.NeutronApiService)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 113, in serve_wsgi
2014-05-16 14:28:36.975 20499 TRACE neutron     LOG.exception(_('Unrecoverable error: please check log '
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 82, in __exit__
2014-05-16 14:28:36.975 20499 TRACE neutron     six.reraise(self.type_, self.value, self.tb)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 106, in serve_wsgi
2014-05-16 14:28:36.975 20499 TRACE neutron     service.start()
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 75, in start
2014-05-16 14:28:36.975 20499 TRACE neutron     self.wsgi_app = _run_wsgi(self.app_name)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 175, in _run_wsgi
2014-05-16 14:28:36.975 20499 TRACE neutron     app = config.load_paste_app(app_name)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 170, in load_paste_app
2014-05-16 14:28:36.975 20499 TRACE neutron     app = deploy.loadapp("config:%s" % config_path, name=app_name)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2014-05-16 14:28:36.975 20499 TRACE neutron     return loadobj(APP, uri, name=name, **kw)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2014-05-16 14:28:36.975 20499 TRACE neutron     return context.create()
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2014-05-16 14:28:36.975 20499 TRACE neutron     return self.object_type.invoke(self)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2014-05-16 14:28:36.975 20499 TRACE neutron     **context.local_conf)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
2014-05-16 14:28:36.975 20499 TRACE neutron     val = callable(*args, **kw)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2014-05-16 14:28:36.975 20499 TRACE neutron     app = loader.get_app(app_name, global_conf=global_conf)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2014-05-16 14:28:36.975 20499 TRACE neutron     name=name, global_conf=global_conf).create()
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2014-05-16 14:28:36.975 20499 TRACE neutron     return self.object_type.invoke(self)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2014-05-16 14:28:36.975 20499 TRACE neutron     **context.local_conf)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
2014-05-16 14:28:36.975 20499 TRACE neutron     val = callable(*args, **kw)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 72, in pipeline_factory
2014-05-16 14:28:36.975 20499 TRACE neutron     app = filter(app)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py", line 1387, in auth_filter
2014-05-16 14:28:36.975 20499 TRACE neutron     return AuthProtocol(app, conf)
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py", line 467, in __init__
2014-05-16 14:28:36.975 20499 TRACE neutron     self.signing_dirname = tempfile.mkdtemp(prefix='keystone-signing-')
2014-05-16 14:28:36.975 20499 TRACE neutron   File "/usr/lib64/python2.7/tempfile.py", line 329, in mkdtemp
2014-05-16 14:28:36.975 20499 TRACE neutron     _os.mkdir(file, 0700)
2014-05-16 14:28:36.975 20499 TRACE neutron OSError: [Errno 13] Permission denied: '/tmp/keystone-signing-Ipk4ys'
2014-05-16 14:28:36.975 20499 TRACE neutron 

Note that other services don't have problem creating tmp directories:
[root@localhost ~]# ll /tmp/
total 4
drwx------. 2 nova   nova     6 May 16 14:11 keystone-signing-1UMWN9
drwx------. 2 nova   nova     6 May 16 14:11 keystone-signing-eB_7Of
drwx------. 2 cinder cinder   6 May 16 14:10 keystone-signing-fgatlb
drwx------. 2 nova   nova     6 May 16 14:11 keystone-signing-MdZcVv
drwx------. 2 cinder cinder   6 May 16 14:10 keystone-signing-wwWk3m


Version-Release number of selected component (if applicable):
[root@localhost ~]# rpm -qa openstack-neutron*
openstack-neutron-2014.1-19.el7ost.noarch
openstack-neutron-openvswitch-2014.1-19.el7ost.noarch

Comment 2 Ihar Hrachyshka 2014-05-16 13:09:02 UTC
Are there any AVC violations in logs?

Comment 3 Martin Magr 2014-05-19 10:52:08 UTC
Ah right, selinux again. Yup there is one:
[para@localhost ~]$ sudo ausearch -m avc | grep neutron
type=SYSCALL msg=audit(1400496218.377:9082): arch=c000003e syscall=83 success=no exit=-13 a0=2fdbd40 a1=1c0 a2=7f33c074bf88 a3=7fff3be58d90 items=0 ppid=1 pid=22546 auid=4294967295 uid=994 gid=993 euid=994 suid=994 fsuid=994 egid=993 sgid=993 fsgid=993 tty=(none) ses=4294967295 comm="neutron-server" exe="/usr/bin/python2.7" subj=system_u:system_r:neutron_t:s0 key=(null)
type=AVC msg=audit(1400496218.377:9082): avc:  denied  { create } for  pid=22546 comm="neutron-server" name="keystone-signing-cWBO58" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir

Comment 4 Miroslav Grepl 2014-05-19 11:35:11 UTC
Could you re-test it in permissive mode?

Comment 5 Miroslav Grepl 2014-05-19 11:36:14 UTC
Actually what does show you

rpm -q selinux-policy-targeted

Comment 6 Martin Magr 2014-05-19 12:57:27 UTC
In permissive mode Neutron server works OK.

[para@localhost ~]$ rpm -q selinux-policy-targeted
selinux-policy-targeted-3.12.1-153.el7.noarch

Comment 7 Miroslav Grepl 2014-05-19 13:17:43 UTC
(In reply to Martin Magr from comment #6)
> In permissive mode Neutron server works OK.
> 
> [para@localhost ~]$ rpm -q selinux-policy-targeted
> selinux-policy-targeted-3.12.1-153.el7.noarch

Sure, I wanted to see AVC msgs.

Comment 8 Ihar Hrachyshka 2014-05-19 13:23:59 UTC
Isn't 'ausearch' output enough?

Comment 9 Miroslav Grepl 2014-05-20 05:59:42 UTC
Yes, it is. 'ausearch' in permissive mode after re-testing.

And what does

$ sesearch -T -s neutron_t -c dir |grep tmp_t

Comment 13 Hugh Brock 2014-06-03 14:06:54 UTC
This should be in last night's puddle

Comment 14 Ofer Blaut 2014-06-05 14:35:57 UTC
Tested on selinux-policy-3.12.1-153.el7_0.10.noarch

Jun  5 10:14:56 puma04 kernel: type=1400 audit(1401952496.719:11): avc:  denied  { create } for  pid=22723 comm="neutron-server" name="keystone-signing-YDi1vz" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir

Comment 15 Miroslav Grepl 2014-06-05 15:03:53 UTC
What does

#sesearch -T -s neutron_t -c dir |grep tmp_t

Comment 18 Lon Hohberger 2014-06-09 22:12:14 UTC
*** Bug 1106296 has been marked as a duplicate of this bug. ***

Comment 19 Lon Hohberger 2014-06-10 14:05:35 UTC
bug 1106296 was a duplicate of this one - Giulio, were you using the newer package?

Comment 24 Lon Hohberger 2014-06-12 18:54:21 UTC
It appears that this was fixed by the updated package from Miroslav:

https://bugzilla.redhat.com/show_bug.cgi?id=1099044#c3

Comment 27 Lon Hohberger 2014-06-18 14:22:17 UTC
This looks like upstream commit 30935facbaf4e1139088076e312070ea95293aa0 - I'll see if it's in the .10 package.

Comment 28 Lon Hohberger 2014-06-18 15:49:34 UTC
The .10 package contains the referenced fixes, yet, the problem persists.  Apparently, rebooting cures the problem.

type=AVC msg=audit(1400496218.377:9082): avc:  denied  { create } for  pid=22546 comm="neutron-server" name="keystone-signing-cWBO58" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir

	Was caused by:
		Missing type enforcement (TE) allow rule.

		You can use audit2allow to generate a loadable module to allow this access.

Audit2allow:

require {
	type neutron_t;
	type tmp_t;
	class dir create;
}

#============= neutron_t ==============
allow neutron_t tmp_t:dir create;

This is interesting.  I wonder if neutron is creating tmp files in the wrong place initially for some reason, so a reboot fixes it.

Comment 29 Lon Hohberger 2014-06-18 15:52:31 UTC
Prior to reboot, a user may get an error like this:

2014-06-10 21:59:43.602 22927 TRACE neutron.service OSError: [Errno 13] Permission denied: '/tmp/keystone-signing-igsY_2'

After reboot, the temp files are in a different location, like this:

/tmp/systemd-private-qsb5vM/tmp/keystone-signing-hFUcLA

... and things work.

Comment 30 Lon Hohberger 2014-06-18 18:35:42 UTC
Except, I can't reproduce the original behavior on a clean install with 7.0.z and the current latest available bits for RHEL OSP - 

[root@localhost tmp]# find /tmp -user neutron
/tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC
/tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC/signing_cert.pem
/tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC/cacert.pem
/tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC/revoked.pem

[root@localhost tmp]# ls -lZ /tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC
-rw-------. neutron neutron system_u:object_r:neutron_tmp_t:s0 cacert.pem
-rw-------. neutron neutron system_u:object_r:neutron_tmp_t:s0 revoked.pem
-rw-------. neutron neutron system_u:object_r:neutron_tmp_t:s0 signing_cert.pem
[root@localhost tmp]# ls -ldZ /tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC
drwx------. neutron neutron system_u:object_r:neutron_tmp_t:s0 /tmp/systemd-private-SItMmm/tmp/keystone-signing-idDdDC

Comment 31 Lon Hohberger 2014-06-18 18:42:29 UTC
[root@localhost tmp]# ausearch -i -m avc
<no matches>

openstack-neutron-2014.1-34.el7ost.noarch
selinux-policy-3.12.1-153.el7_0.10.noarch
openstack-selinux-0.5.2-2.el7ost.noarch

Comment 33 Lon Hohberger 2014-06-18 19:48:52 UTC
Packstack in the collection you tested:

./packstack/puppet/templates/postscript.pp:exec { 'update-selinux-policy':
./packstack/puppet/templates/postscript.pp:    command => "yum update -y selinux-policy-targeted"

So, I think the problem is this:

1) Install RHEL 7.0
2) Use packstack to deploy

At the end of installation, *after* tmpfiles are used by neutron (and presumably denied until then) - so, a reboot works.

Now, with current packstack:

./packstack/puppet/templates/prescript.pp:# We don't have openstack-selinux package for Fedora
./packstack/puppet/templates/prescript.pp:  package{ 'openstack-selinux':

The openstack-selinux package depends on selinux-policy .10 release, meaning that it's installed before neutron is started.  So, this should not occur any more.  I'm trying to reproduce, however.

Comment 34 Lon Hohberger 2014-06-18 19:49:55 UTC
(In reply to Lon Hohberger from comment #33)
> At the end of installation, *after* tmpfiles are used by neutron (and
> presumably denied until then) - so, a reboot works.
> 

That is - at the end of installation, selinux-policy is updated by packstack.

Comment 35 Lon Hohberger 2014-06-18 20:12:12 UTC
Reproduced.  This is indeed the problem, and it will be resolved because openstack-selinux depends on the 7.0.z selinux-policy, which has the fixes Miroslav noted.

Comment 36 Ofer Blaut 2014-06-22 10:32:23 UTC
verified 

openstack-packstack-2014.1.1-0.26.dev1157.el7ost.noarch
openstack-packstack-puppet-2014.1.1-0.26.dev1157.el7ost.noarch

selinux-policy-3.12.1-153.el7_0.10.noarch
openstack-selinux-0.5.2-2.el7ost.noarch

Comment 38 errata-xmlrpc 2014-07-08 15:13:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0845.html


Note You need to log in before you can comment on or make changes to this bug.