Bug 1320043

Summary:

rootwrap-daemon can't start after reboot due to AVC denial

Product:

[Community] RDO

Reporter:

Eran Kuris <ekuris>

Component:

openstack-selinux

Assignee:

Lon Hohberger <lhh>

Status:

CLOSED DUPLICATE

QA Contact:

Ofer Blaut <oblaut>

Severity:

urgent

Docs Contact:

Priority:

unspecified

Version:

trunk

CC:

chrisw, ekuris, jlibosva, nyechiel, srevivo, yrachman

Target Milestone:

---

Target Release:

trunk

Hardware:

Unspecified

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2016-09-01 08:36:19 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
ovs firewall	none

Description Eran Kuris 2016-03-22 08:23:41 UTC

Created attachment 1138919 [details]
ovs firewall

Description of problem:
Install RDO-Mitaka (milestone 3) with packstack on CentOS.
After installation changed OVS rpm to version 2.5 .
after restart the services we  can see that   neutron-openvswitch-agent is in failed status.

In log file : 
2016-03-21 13:59:03.517 4548 ERROR neutron Traceback (most recent call last):
2016-03-21 13:59:03.517 4548 ERROR neutron   File "/usr/bin/neutron-rootwrap-daemon", line 10, in <module>
2016-03-21 13:59:03.517 4548 ERROR neutron     sys.exit(daemon())
2016-03-21 13:59:03.517 4548 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_rootwrap/cmd.py", line 57, in daemon
2016-03-21 13:59:03.517 4548 ERROR neutron     return main(run_daemon=True)
2016-03-21 13:59:03.517 4548 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_rootwrap/cmd.py", line 98, in main
2016-03-21 13:59:03.517 4548 ERROR neutron     daemon_mod.daemon_start(config, filters)
2016-03-21 13:59:03.517 4548 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_rootwrap/daemon.py", line 98, in daemon_start
2016-03-21 13:59:03.517 4548 ERROR neutron     server = manager.get_server()
2016-03-21 13:59:03.517 4548 ERROR neutron   File "/usr/lib64/python2.7/multiprocessing/managers.py", line 493, in get_server
2016-03-21 13:59:03.517 4548 ERROR neutron     self._authkey, self._serializer)
2016-03-21 13:59:03.517 4548 ERROR neutron   File "/usr/lib64/python2.7/multiprocessing/managers.py", line 162, in __init__
2016-03-21 13:59:03.517 4548 ERROR neutron     self.listener = Listener(address=address, backlog=16)
2016-03-21 13:59:03.517 4548 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_rootwrap/jsonrpc.py", line 66, in __init__
2016-03-21 13:59:03.517 4548 ERROR neutron     self._socket.bind(address)
2016-03-21 13:59:03.517 4548 ERROR neutron   File "/usr/lib64/python2.7/socket.py", line 224, in meth
2016-03-21 13:59:03.517 4548 ERROR neutron     return getattr(self._sock,name)(*args)
2016-03-21 13:59:03.517 4548 ERROR neutron socket.error: [Errno 13] Permission denied


Version-Release number of selected component (if applicable):
[root@puma15 ~(keystone_admin)]# uname -a
Linux puma15.scl.lab.tlv.redhat.com 4.5.0-1.el7.elrepo.x86_64 #1 SMP Mon Mar 14 10:24:58 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@puma15 ~(keystone_admin)]# rpm -qa |grep openvswitch
python-openvswitch-2.5.0-2.el7.noarch
openstack-neutron-openvswitch-8.0.0.0b4-0.20160304174813.0ae20a3.el7.centos.noarch
openvswitch-2.5.0-2.el7.x86_64
[root@puma15 ~(keystone_admin)]# rpm -qa |grep neutron
openstack-neutron-8.0.0.0b4-0.20160304174813.0ae20a3.el7.centos.noarch
openstack-neutron-ml2-8.0.0.0b4-0.20160304174813.0ae20a3.el7.centos.noarch
python-neutron-8.0.0.0b4-0.20160304174813.0ae20a3.el7.centos.noarch
openstack-neutron-openvswitch-8.0.0.0b4-0.20160304174813.0ae20a3.el7.centos.noarch
openstack-neutron-common-8.0.0.0b4-0.20160304174813.0ae20a3.el7.centos.noarch
python-neutronclient-4.1.2-0.20160304195803.5d28651.el7.centos.noarch
python-neutron-lib-0.0.3-0.20160227020344.999828a.el7.centos.noarch
openstack-neutron-metering-agent-8.0.0.0b4-0.20160304174813.0ae20a3.el7.centos.noarch


How reproducible:
always

Steps to Reproduce:
1.Install RDO-Mitaka with packstack in CentOS 
2. rpm -e --nodeps openvswitch-2.4.0-1.el7.x86_64
3. rpm -e --nodeps python-openvswitch-2.4.0-1.el7.noarch
4. rpm -ivh   http://cbs.centos.org/kojifiles/packages/openvswitch/2.5.0/2.el7/noarch/python-openvswitch-2.5.0-2.el7.noarch.rpm 
5. rpm -ivh http://cbs.centos.org/kojifiles/packages/openvswitch/2.5.0/2.el7/x86_64/openvswitch-2.5.0-2.el7.x86_64.rpm 
6. reboot host 
7. openstack-status --- neutron ovs agent is down

Actual results:
neutron ovs agent is down

Expected results:

all services in active no errors 
Additional info:
change Selinux to disable fixed the issue

Comment 1 Jakub Libosvar 2016-03-22 08:39:13 UTC

Can you please switch to permissive mode, kill rootwrap daemon and start ovs agent again? This way we'll collect all denials that rootwrap daemon produces.

Comment 2 Eran Kuris 2016-03-22 10:08:11 UTC

I get some more avc in audit log : 
type=AVC msg=audit(1458637620.131:3164): avc:  denied  { create } for  pid=6988 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutr85 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458637776.467:3218): avc:  denied  { create } for  pid=7r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458637856.473:3249): avc:  denied  { create } for  pid=7149 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:systemile permissive=0
type=AVC msg=audit(1458637923.473:3273): avc:  denied  { create } for  pid=7194 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_c:  denied  { create } for  pid=7246 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458638014.464:3303): arootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458638075.119:3327): avc:  denied  { create } for  pid=7282 comm="neutron-rootwra" name=s=sock_file permissive=0
type=AVC msg=audit(1458638136.491:3351): avc:  denied  { create } for  pid=7316 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tcla8638216.498:3375): avc:  denied  { create } for  pid=7371 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1426 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458638296.507:3405): avc:  denied  { create } for  pid=7n_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458638376.513:3429): avc:  denied  { create } for  pid=7462 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutr-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458638530.168:3495): avc:  denied  { create } for  pid=7576 comm="neutrocontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458638590.825:3519): avc:  denied  { create } for  pid=7628 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 
type=AVC msg=audit(1458638656.533:3543): avc:  denied  { create } for  pid=7658 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=  { create } for  pid=7696 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458638742.904:3573): avc:  deniecontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458638816.547:3597): avc:  denied  { create } for  pid=7753 comm="neutron-rootwra" name="rootwrap.sock" 
type=AVC msg=audit(1458638894.855:3621): avc:  denied  { create } for  pid=7809 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=31mavc:  denied  { create } for  pid=7837 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458638976.561:3651): [01.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458639046.524:3682): avc:  denied  { create } for  pid=7894 comm="neutron-rootwra" name="rootwra:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458639107.182:3706): avc:  denied  { create } for  pid=7943 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_8639176.577:3730): avc:  denied  { create } for  pid=8001 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(14rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458639258.806:3804): avc:  denied  { create } for  pid=8116 comm="neutron-rootwra" name=context=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458639336.590:3828): avc:  denied  { create } for  pid=8155 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 
type=AVC msg=audit(1458639410.436:3852): avc:  denied  { create } for  pid=8205 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=  denied  { create } for  pid=8260 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458639496.598:3882): avc.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458639562.086:3906): avc:  denied  { create } for  pid=8296 comm="neutron-rootwra" name="rootwras0 tclass=sock_file permissive=0
type=AVC msg=audit(1458639622.736:3937): avc:  denied  { create } for  pid=8337 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t606:3961): avc:  denied  { create } for  pid=8399 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458639713"neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458639776.614:3991): avc:  denied  { create } for  pid=8474 commcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458639856.620:4015): avc:  denied  { create } for  pid=8506 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 udit(1458639926.203:4039): avc:  denied  { create } for  pid=8542 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=82 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458640016.627:4069): avc:  denied  { create } for  pid=8r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458640078.053:4093): avc:  denied  { create } for  pid=8654 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:systems=sock_file permissive=0
type=AVC msg=audit(1458640138.903:4117): avc:  denied  { create } for  pid=8682 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tcla31mavc:  denied  { create } for  pid=8727 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458640230.068:4154): [01" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458640296.654:4186): avc:  denied  { create } for  pid=8787 comm="neutron-rootwr:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458640376.660:4210): avc:  denied  { create } for  pid=8854 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_udit(1458640442.564:4234): avc:  denied  { create } for  pid=8882 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=  pid=8910 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458640533.554:4264): avc:  denied  { create } for:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1458640594.210:4288): avc:  denied  { create } for  pid=8966 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:systemissive=0
type=AVC msg=audit(1458640656.686:4312): avc:  denied  { create } for  pid=9006 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file per denied  { create } for  pid=9110 comm="neutron-rootwra" name="rootwrap.sock" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_tmp_t:s0 tclass=sock_file permissive=1
type=AVC msg=audit(1458640736.695:4350): avc: path="/run/netns/qrouter-63deb918-e054-47fd-a2f8-dac1548acf18" dev="nsfs" ino=4026532504 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
type=AVC msg=audit(1458640737.582:4355): avc[0 msg='avc:  received setenforce notice (enforcing=0)  exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
type=AVC msg=audit(1458640835.553:4395): avc:  denied  { create } for  pid=9514 comm="neutron-ro

Comment 3 Nir Yechiel 2016-08-31 19:23:14 UTC

Yariv,

Is this one still valid?

Thanks,
Nir

Comment 4 Yariv 2016-08-31 21:37:23 UTC

We are verifying the following bug: https://bugzilla.redhat.com/show_bug.cgi?id=1278495

It seems related, we will update once it is verified

Comment 5 Nir Yechiel 2016-09-01 08:36:19 UTC


*** This bug has been marked as a duplicate of bug 1278495 ***