Bug 1402032

Summary: [fdProd] RHOS 10 instance gets error state with openvswitch-2.5.0-22 installed on overcloud
Product: Red Hat OpenStack Reporter: Maxim Babushkin <mbabushk>
Component: openstack-selinuxAssignee: Ryan Hallisey <rhallise>
Status: CLOSED NOTABUG QA Contact: Udi Shkalim <ushkalim>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 10.0 (Newton)CC: aconole, aloughla, amuller, atelang, fbaudin, fleitner, lhh, mbabushk, mburns, mgrepl, nyechiel, oblaut, rkhan, skramaja, srevivo, vchundur, yrachman
Target Milestone: asyncKeywords: ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-07 13:11:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
/var/log/neutron/openvswitch-agent.log none

Description Maxim Babushkin 2016-12-06 15:38:31 UTC
Created attachment 1228609 [details]
/var/log/neutron/openvswitch-agent.log

Description of problem:
RHOS10 OVS DPDK unable to boot an instance with openvswitch-2.5.0-22.

I have installed manually openvswitch-2.5.0-22 within overcloud-full.qcow2 image.
Deployed an overcloud and tried to boot an instance. Get error state.
DPDK port binded successfully.

Version-Release number of selected component (if applicable):
RHOS10
openvswitch-2.5.0-22

Steps to Reproduce:
1. Install openvswitch-2.5.0-22 within overcloud-full qcow2 image.
2. Deploy an overcloud.
3. Boot an instance.

Actual results:
Instance enters an error state.

Expected results:
Instance should boot successfully.

Additional info:
The openvswitch-agent error log attached.

Comment 1 Aaron Conole 2016-12-06 15:44:27 UTC
Can you attach an sosreport from the system?  I want to see what the state of openvswitch is at the time of error.  Your guest agent seems to indicate an error serializing something to the database.

Comment 3 Aaron Conole 2016-12-06 16:58:47 UTC
I see errors like the following:

type=AVC msg=audit(1481038212.934:103): avc:  denied  { execute } for  pid=3823 comm="neutron-rootwra" name="ovs-vsctl" dev="sda2" ino=10738413 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file

So, neutron-rootwrap is getting denials trying to run ovs-vsctl.  Perhaps there's some missing neutron selinux rules?

Comment 4 Maxim Babushkin 2016-12-06 17:05:40 UTC
We have verified selinux policy for openvswitch-2.5-0.14.

Openvswitch-2.5.0-22 and 2.5.0-14 have some changes between versions.
Maybe, existing policy does not covering these changes.

But, when I run manual update of the openvswitch from 2.5.0-14 to 2.5.0-22 in the existing environment, instance was able to boot successfully.

Comment 5 Maxim Babushkin 2016-12-06 19:26:04 UTC
I will verify it is a selinux bug, collect the alerts and involve selinux team.

Comment 6 Franck Baudin 2016-12-07 08:56:59 UTC
If you install with 2.5.0-22 from scratch, do you have the same issue?

Comment 7 Maxim Babushkin 2016-12-07 09:06:06 UTC
It seems that the major change in ovs 2.5.0-22 not covered by selinux policy we have validated in 2.5.0-14 version.
Currently, verifying it.

Comment 8 Maxim Babushkin 2016-12-07 13:10:40 UTC
Not a bug.
It seems that during the manual installation of ovs 2.5.0-22 on the overcloud-full image with virt-customize, something went wrong.

Now, as with the latest puddle, overcloud image comes with ovs 2.5.0-22, I verified twice that overcloud deploy finish successfully, and an instance with dpdk is able to boot and get dhcp allocation without any issue.