Description of problem: Please merge https://review.openstack.org/#/c/397062/ from upstream into OSP. This fixes bug https://bugs.launchpad.net/neutron/+bug/1590845 From the upstream bug: When we programmatically attach subnets and/or ports to routers(we attach 1 interface every 60 seconds), some report back stuck in the BUILD state. Take this interface, for example: neutron port-show 98b55b89-a002-496f-a5d4-8de598613da8 +-----------------------+--------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+--------------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | dn3usoskctl03_neutron_agents_container-e64e37d6 | | binding:profile | {} | | binding:vif_details | {"port_filter": true} | | binding:vif_type | bridge | | binding:vnic_type | normal | | device_id | 5838c5de-e87a-4e5e-b61f-a3f068fa7726 | | device_owner | network:router_interface | | dns_assignment | {"hostname": "host-10-169-160-1", "ip_address": "10.169.160.1", "fqdn": "host-10-169-160-1.openstacklocal."} | | dns_name | | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "bc3a8d37-6cd7-4d57-b0c9-2b35743b0a0b", "ip_address": "10.169.160.1"} | | id | 98b55b89-a002-496f-a5d4-8de598613da8 | | mac_address | fa:16:3e:b9:7a:1d | | name | | | network_id | 535c3336-202c-4dab-b517-2232c4ce1481 | | security_groups | | | status | BUILD | | tenant_id | 3ccf712795c44edcbc8ffcc331a59853 | +-----------------------+--------------------------------------------------------------------------------------------------------------+ It's reporting itself in the BUILD state, but when I check the router namespace, it's linux networking component counter part seems to be functioning just fine: 8: qr-98b55b89-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:b9:7a:1d brd ff:ff:ff:ff:ff:ff inet 10.169.160.1/23 scope global qr-98b55b89-a0 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feb9:7a1d/64 scope link valid_lft forever preferred_lft forever I can even ping the address with no problem once i open up the security group rules. --------------------------- Just to add to this, we have seen that if the port is in BUILD state and l2pop is enabled, the FDB and ARP tables will not updated across nodes. This results in lack of connectivity to/from the qr ports of the router. The fix in this case is to convert the router back to standalone. ----------------------------- We are also seeing this issue in stable/mitaka. Same as James Denton mentioned in comment #9 when router interface is stuck in BUILD, ARP and FDB tables do not getting updated and it causes connectivity issues. We are able to reproduce the issue by failing over the node where the master router is hosted.
Hi Assaf In upstream, this fix is not yet included in mitaka tag. So I will cherry pic and release a build. Thanks Anil
I would like to get steps to reproduces for this issue .
The steps are in the upstream bug https://bugs.launchpad.net/neutron/+bug/1590845 This happens when using linux bridge, not OVS bridge but for the sake of completeness, here are the notes that I took when I deployed my reproducer: convert an openstack director deployment to linux bridge: compute: 37 subscription-manager repos --disable=* 38 subscription-manager repos --enable=rhel-7-server-rpms 39 subscription-manager repos --enable=rhel-7-server-rh-common-rpms 40 subscription-manager repos --enable=rhel-7-server-openstack-9-rpms 41 subscription-manager repos --enable=rhel-7-server-extras-rpms 42 yum install openstack-neutron openstack-neutron-ml2 openstack-neutron-linuxbridge ebtables 43 vi /etc/neutron/plugins/ml2/linuxbridge_agent.ini 44 brctl show 45 vi /etc/neutron/plugins/ml2/linuxbridge_agent.ini 46 ip a | grep 172.16.0 47 vi /etc/neutron/plugins/ml2/linuxbridge_agent.ini 48 systemctl list-units | grep openv 49 systemctl stop neutron-openvswitch-agent.service 50 systemctl disable neutron-openvswitch-agent.service. 51 systemctl disable neutron-openvswitch-agent.service 52 systemctl enable openstack-linuxbridge-agent 53 systemctl list-unit-files | grep linuxb 54 systemctl enable neutron-linuxbridge-agent.service 55 systemctl start neutron-linuxbridge-agent.service 56 ovs-vsctl show 57 ovs-vsctl del-br br-int 58 ovs-vsctl del-br br-ex 59 ovs-vsctl del-br br-tun 60 ovs-vsctl show 61 systemctl stop openvswitch 62 systemctl disable openvswitch 63 systemctl list-units 64 systemctl list-units | grep openv /etc/neutron/l3_agent.ini /etc/neutron/dhcp_agent.ini controllers: 20 subscription-manager repos --disable=* 21 subscription-manager repos --enable=rhel-7-server-rpms 22 subscription-manager repos --enable=rhel-7-server-rh-common-rpms 23 subscription-manager repos --enable=rhel-7-server-openstack-9-rpms 24 subscription-manager repos --enable=rhel-7-server-extras-rpms 25 yum install openstack-neutron openstack-neutron-ml2 openstack-neutron-linuxbridge ebtables 26 vi /etc/neutron/plugins/ml2/linuxbridge_agent.ini 27 brctl show 28 vi /etc/neutron/plugins/ml2/linuxbridge_agent.ini 29 ip a | grep 172.16.0 30 vi /etc/neutron/plugins/ml2/linuxbridge_agent.ini 31 pcs status 32 pcs resource disable neutron-openvswitch-agent-clone 33 cibadmin -Q | grep neutron-openvswitch-agent 34 cibadmin --query > tmp.xml 35 vi tmp.xml ==> replace neutron-openvswitch-agent-clone with neutron-linuxbridge-agent-clone ==> replace neutron-ovs-cleanup with neutron-linuxbridge-cleanup 36 cibadmin --replace --xml-file tmp.xml 37 pcs status 38 pcs status | grep linuxb 39 pcs status | grep linuxb -A2 40 pcs resource enable neutron-linuxbridge-agent-clone 41 pcs status neutron-linuxbridge-agent-clone 42 pcs resource show neutron-linuxbridge-agent-clone 43 systemctl list-units | grep linuxb 44 systemctl status neutron-linuxbridge-agent.service 45 systemctl stop openvswitch 46 ovs-vsctl show 47 systemctl start openvswitch 48 ovs-vsctl show 49 ovs-vsctl del-br br-int 50 ovs-vsctl del-br br-tun 51 ovs-vsctl del-br br-ex 52 ovs-vsctl show 53 systemctl stop openvswitch 54 systemctl disable openvswitch 55 pcs status 56 pcs resource cleanup 57 pcs resource cleanup --force /etc/neutron/l3_agent.ini /etc/neutron/dhcp_agent.ini follow http://docs.openstack.org/mitaka/install-guide-rdo/neutron-controller-install-option2.html in linux bridge, external network CANNOT be the same vlanid as controller external network --------------- nova floating-ip-list | grep 10.0 | awk '{print $4}' | xargs -I {} nova floating-ip-delete {} neutron router-interface-delete router `neutron subnet-list | grep private | awk '{print $2}'` neutron router-gateway-clear router neutron router-delete router ----------------------------------------------------------------- neutron router-create router --ha false or neutron router-create router --ha true neutron router-interface-add router `neutron subnet-list | grep private | awk '{print $2}'` neutron router-gateway-set router provider1 nova floating-ip-create provider1 nova floating-ip-associate cirros-test1 `nova floating-ip-list | grep 10.0 | awk '{print $4}'` ----------------------------------------------------------------- neutron port-list | awk '/fa:16/ {print $2}' | xargs -I {} neutron port-show {} | egrep -i 'status|ip' ------------------------------------------------------------------ neutron port-list | awk '/fa:16/ {print $2}' | xargs -I {} neutron port-show {} | egrep -i 'status|ip' | grep -i build | wc -l ------------------------------------------------------------------ #!/bin/bash for i in {1 .. 10};do nova floating-ip-list | grep 10.0 | awk '{print $4}' | xargs -I {} nova floating-ip-delete {} neutron router-interface-delete router `neutron subnet-list | grep private | awk '{print $2}'` neutron router-gateway-clear router neutron router-delete router neutron router-create router --ha true neutron router-interface-add router `neutron subnet-list | grep private | awk '{print $2}'` neutron router-gateway-set router provider1 nova floating-ip-create provider1 nova floating-ip-associate cirros-test1 `nova floating-ip-list | grep 10.0 | awk '{print $4}'` neutron port-list | awk '/fa:16/ {print $2}' | xargs -I {} neutron port-show {} | egrep -i 'status|ip' | grep -i build | wc -l done
Thanks Andreas Karis. @ekuris, You need linux bridge agent to reproduce this. Please use steps explained by Andreas Karis.