Hide Forgot
Created attachment 1335154 [details] Controller logs Description of problem: Neutron service is unavailable after deploying OSP 12 with OVN (HA) enabled. Version-Release number of selected component (if applicable): OSP 12 PUDDLE 2017-09-28.1 TOPOLOGY: 3 controllers, 2 computes How reproducible: 1) Run deploy command with neutron-ml2-ovn-ha.yaml templates included #!/bin/bash timeout 100m openstack overcloud deploy \ --templates /usr/share/openstack-tripleo-heat-templates \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ml2-ovn-ha.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/extra_templates.yaml \ -e /home/stack/virt/docker-images.yaml \ -e /home/stack/virt/docker-images-ceph.yaml \ -e /home/stack/virt/nodes_data.yaml \ --log-file overcloud_deployment_40.log 2) After deploy command ssh to the undercloud and run: $ source overcloudrc $ openstack network list Actual results: (overcloud) [stack@undercloud-0 ~]$ openstack network list Unable to parse endpoints for network Expected results: List of networks to be listed. Additional info: Not sure what logs to provide. you can find logs snapshot here: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/OSPD-Customized-Deployment-virt/1201/artifact/ Attaching logs from controller-0. [heat-admin@controller-0 ~]$ tail /var/log/openvswitch/ovn-controller.log-20171006 2017-10-06T07:55:09.639Z|23151|reconnect|INFO|tcp:172.17.1.11:6642: waiting 8 seconds before reconnect 2017-10-06T07:55:17.647Z|23152|reconnect|INFO|tcp:172.17.1.11:6642: connecting... 2017-10-06T07:55:17.647Z|23153|reconnect|INFO|tcp:172.17.1.11:6642: connection attempt failed (Connection refused) 2017-10-06T07:55:17.647Z|23154|reconnect|INFO|tcp:172.17.1.11:6642: waiting 8 seconds before reconnect 2017-10-06T07:55:25.650Z|23155|reconnect|INFO|tcp:172.17.1.11:6642: connecting... 2017-10-06T07:55:25.650Z|23156|reconnect|INFO|tcp:172.17.1.11:6642: connection attempt failed (Connection refused) 2017-10-06T07:55:25.650Z|23157|reconnect|INFO|tcp:172.17.1.11:6642: waiting 8 seconds before reconnect 2017-10-06T07:55:33.653Z|23158|reconnect|INFO|tcp:172.17.1.11:6642: connecting... 2017-10-06T07:55:33.653Z|23159|reconnect|INFO|tcp:172.17.1.11:6642: connection attempt failed (Connection refused) 2017-10-06T07:55:33.653Z|23160|reconnect|INFO|tcp:172.17.1.11:6642: waiting 8 seconds before reconnect
From the ovsdb-server logs, I clearly see that ovsdb-servers (NB and SB) are not able to open the tcp sockets 6641 and 6642 and see the below error messages. ******** 2017-10-05T14:41:10.412Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server-nb.log 2017-10-05T14:41:10.415Z|00002|socket_util|ERR|6641:172.17.1.11: bind: Permission denied 2017-10-05T14:41:10.415Z|00003|ovsdb_jsonrpc_server|ERR|ptcp:6641:172.17.1.11: listen failed: Permission denied 2017-10-05T14:41:10.418Z|00004|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.7.2 2017-10-05T14:41:10.418Z|00005|socket_util|ERR|6641:172.17.1.11: bind: Permission denied ********** Looks like its an selinux issue. Similar issue is reported here for port 6639 - https://bugzilla.redhat.com/show_bug.cgi?id=1498921
In the /var/log/audit/audit.log, I can see the below error messages which clearly indicates that its selinux issue. ********* type=AVC msg=audit(1507214470.415:1155): avc: denied { name_bind } for pid=72402 comm="ovsdb-server" src=6641 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket type=SYSCALL msg=audit(1507214470.415:1155): arch=c000003e syscall=49 success=no exit=-13 a0=f a1=7ffcbf1e3240 a2=10 a3=7ffcbf1e3238 items=0 ppid=72401 pid=72402 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovsdb-server" exe="/usr/sbin/ovsdb-server" subj=system_u:system_r:openvswitch_t:s0 key=(null) type=PROCTITLE msg=audit(1507214470.415:1155): proctitle=6F767364622D736572766572002D2D646574616368002D2D6D6F6E69746F72002D76636F6E736F6C653A6F6666002D2D6C6F672D66696C653D2F7661722F6C6F672F6F70656E767377697463682F6F767364622D7365727665722D6E622E6C6F67002D2D72656D6F74653D70756E69783A2F7661722F72756E2F6F70656E7673 type=AVC msg=audit(1507214470.418:1156): avc: denied { name_bind } for pid=72402 comm="ovsdb-server" src=6641 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket .... proctitle=6F767364622D736572766572002D2D646574616368002D2D6D6F6E69746F72002D76636F6E736F6C653A6F6666002D2D6C6F672D66696C653D2F7661722F6C6F672F6F70656E767377697463682F6F767364622D7365727665722D6E622E6C6F67002D2D72656D6F74653D70756E69783A2F7661722F72756E2F6F70656E7673 type=AVC msg=audit(1507214470.444:1163): avc: denied { name_bind } for pid=72411 comm="ovsdb-server" src=6642 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket type=SYSCALL msg=audit(1507214470.444:1163): arch=c000003e syscall=49 success=no exit=-13 a0=10 a1=7fffc0c63cb0 a2=10 a3=7fffc0c63ca8 items=0 ppid=72410 pid=72411 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovsdb-server" exe="/usr/sbin/ovsdb-server" subj=system_u:system_r:openvswitch_t:s0 key=(null) type=PROCTITLE msg=audit(1507214470.444:1163): ********************
Wanna take a look at this? I have a recollection that SELinux policies for OVS/OVN are in the OVS repo as opposed to openstack-selinux.
If I recall correctly, port assignment needs to be done in a %post or it can conflict with SELinux base policies. So, I'll add 6641 and 6642 to openstack-selinux as well. https://github.com/redhat-openstack/openstack-selinux/commit/1258d6cf607cf64ba521b6b1ecfa35029d458d6c Up to you if it needs additional parts in ovs repos.
fix verified : rpm -qa | grep selinux selinux-policy-3.13.1-166.el7_4.5.noarch ceph-selinux-10.2.7-48.el7cp.x86_64 libselinux-2.5-11.el7.x86_64 selinux-policy-targeted-3.13.1-166.el7_4.5.noarch libselinux-utils-2.5-11.el7.x86_64 openstack-selinux-0.8.11-0.20171013192233.ce13ba7.el7ost.noarch libselinux-ruby-2.5-11.el7.x86_64 libselinux-python-2.5-11.el7.x86_64 container-selinux-2.28-1.git85ce147.el7.noarch [root@controller-0 ~]# cat /etc/yum.repos.d/latest-installed 12 -p 2017-10-24.4
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462