I am attempting to add a second "external" network to an OSP 13 environment. I've added the bridge to the deployment via the following environment parameters: NeutronBridgeMappings: "datacentre:br-ex,sahara:br-sahara" NeutronFlatNetworks: "datacenre,sahara" This puts the expected configuration into /var/lib/config-data/neutron/etc/neutron/.... On the controllers: .../etc/neutron/plugins/ml2/ml2_conf.ini:flat_networks=datacenre,sahara .../etc/neutron/plugins/ml2/openvswitch_agent.ini:bridge_mappings=datacentre:br-ex,sahara:br-sahara On the network hosts: .../etc/neutron/plugins/ml2/openvswitch_agent.ini:bridge_mappings=datacentre:br-ex,sahara:br-sahara I've created the external network and subnet: [lars@rocket ~]$ openstack network show sahara-access +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2018-11-28T13:27:53Z | | description | | | dns_domain | None | | id | 2ac07a0f-5428-449a-9d8f-e93ac680d2cb | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | location | None | | mtu | 9050 | | name | sahara-access | | port_security_enabled | True | | project_id | 8a6707ab138b468ea32dd259507732d1 | | provider:network_type | flat | | provider:physical_network | sahara | | provider:segmentation_id | None | | qos_policy_id | None | | revision_number | 9 | | router:external | External | | segments | None | | shared | False | | status | ACTIVE | | subnets | 8c5b00c5-b08c-47e7-8414-9251e1df5346 | | tags | | | updated_at | 2018-11-28T13:41:15Z | +---------------------------+--------------------------------------+ [lars@rocket ~]$ openstack subnet show sahara-access-subnet +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 10.251.0.50-10.251.255.254 | | cidr | 10.251.0.0/16 | | created_at | 2018-11-28T13:41:15Z | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | None | | host_routes | | | id | 8c5b00c5-b08c-47e7-8414-9251e1df5346 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | location | None | | name | sahara-access-subnet | | network_id | 2ac07a0f-5428-449a-9d8f-e93ac680d2cb | | project_id | 8a6707ab138b468ea32dd259507732d1 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2018-11-28T13:41:15Z | +-------------------+--------------------------------------+ And a router: [lars@rocket ~]$ openstack router show sahara-router +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2018-11-28T13:41:21Z | | description | | | distributed | None | | external_gateway_info | {"network_id": "2ac07a0f-5428-449a-9d8f-e93ac680d2cb", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "8c5b00c5-b08c-47e7-8414-9251e1df5346", "ip_address": "10.251.0.50"}]} | | flavor_id | None | | ha | None | | id | 43328981-1027-4c03-8159-f1bbee8ed2ff | | interfaces_info | [{"subnet_id": "306ee68c-ad58-4bbe-86e5-a46326b26f82", "ip_address": "10.0.0.24", "port_id": "c024d5f4-bdbf-4249-bd6e-eeb894689fb9"}] | | location | None | | name | sahara-router | | project_id | 4bb09e3a56cd451b9d260ad6c111fd96 | | revision_number | 8 | | routes | | | status | ACTIVE | | tags | | | updated_at | 2018-11-28T13:42:36Z | +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Things look like they are plumbed into OVS correct. Here's the corresponding router namespace from one of our network hosts: [root@neu-17-11-nc2 ~]# ip netns exec qrouter-43328981-1027-4c03-8159-f1bbee8ed2ff ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 58: ha-895752ed-6b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:e9:9c:48 brd ff:ff:ff:ff:ff:ff inet 169.254.192.17/18 brd 169.254.255.255 scope global ha-895752ed-6b valid_lft forever preferred_lft forever inet 169.254.0.2/24 scope global ha-895752ed-6b valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fee9:9c48/64 scope link valid_lft forever preferred_lft forever 63: qr-c024d5f4-bd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:61:36:50 brd ff:ff:ff:ff:ff:ff inet 10.0.0.24/24 scope global qr-c024d5f4-bd valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe61:3650/64 scope link nodad valid_lft forever preferred_lft forever 68: qg-26f86c24-c2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:ff:82:86 brd ff:ff:ff:ff:ff:ff inet 10.251.0.50/16 scope global qg-26f86c24-c2 valid_lft forever preferred_lft forever inet 10.251.0.62/32 scope global qg-26f86c24-c2 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feff:8286/64 scope link nodad valid_lft forever preferred_lft forever And here's what openvswitch looks like: [root@neu-17-11-nc2 ~]# ovs-vsctl show a5576322-1314-45fa-b88d-65f2e4cf85d9 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "vxlan-ac10400d" Interface "vxlan-ac10400d" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.13"} Port "vxlan-ac104010" Interface "vxlan-ac104010" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.16"} Port "vxlan-ac10401d" Interface "vxlan-ac10401d" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.29"} Port "vxlan-ac10401f" Interface "vxlan-ac10401f" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.31"} Port "vxlan-ac10400a" Interface "vxlan-ac10400a" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.10"} Port "vxlan-ac104011" Interface "vxlan-ac104011" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.17"} Port "vxlan-ac104019" Interface "vxlan-ac104019" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.25"} Port "vxlan-ac10400c" Interface "vxlan-ac10400c" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.12"} Port "vxlan-ac104017" Interface "vxlan-ac104017" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.23"} Port "vxlan-ac10400e" Interface "vxlan-ac10400e" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.14"} Port "vxlan-ac104022" Interface "vxlan-ac104022" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.34"} Port "vxlan-ac104026" Interface "vxlan-ac104026" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.38"} Port "vxlan-ac104012" Interface "vxlan-ac104012" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.18"} Port "vxlan-ac104016" Interface "vxlan-ac104016" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.22"} Port "vxlan-ac10401a" Interface "vxlan-ac10401a" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.26"} Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port br-tun Interface br-tun type: internal Port "vxlan-ac10400b" Interface "vxlan-ac10400b" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.11"} Port "vxlan-ac104014" Interface "vxlan-ac104014" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.20"} Port "vxlan-ac104015" Interface "vxlan-ac104015" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.21"} Port "vxlan-ac104013" Interface "vxlan-ac104013" type: vxlan options: {df_default="true", in_key=flow, local_ip="172.16.64.32", out_key=flow, remote_ip="172.16.64.19"} Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "tapcc392b13-8f" tag: 6 Interface "tapcc392b13-8f" type: internal Port "tap34f6364d-56" tag: 4 Interface "tap34f6364d-56" type: internal Port "qr-c024d5f4-bd" tag: 7 Interface "qr-c024d5f4-bd" type: internal Port "ha-13d7dfc1-a9" tag: 9 Interface "ha-13d7dfc1-a9" type: internal Port int-br-ex Interface int-br-ex type: patch options: {peer=phy-br-ex} Port "tapb355789b-17" tag: 5 Interface "tapb355789b-17" type: internal Port "qg-26f86c24-c2" tag: 14 Interface "qg-26f86c24-c2" type: internal Port "qr-0e9a51d1-49" tag: 4 Interface "qr-0e9a51d1-49" type: internal Port "ha-da9ee1f6-a8" tag: 8 Interface "ha-da9ee1f6-a8" type: internal Port "qg-f921a854-20" tag: 12 Interface "qg-f921a854-20" type: internal Port "qr-860aa65b-20" tag: 3 Interface "qr-860aa65b-20" type: internal Port "qg-b1c270f1-e3" tag: 12 Interface "qg-b1c270f1-e3" type: internal Port "tap698ccfa4-c7" tag: 17 Interface "tap698ccfa4-c7" type: internal Port "qr-225e951a-b4" tag: 6 Interface "qr-225e951a-b4" type: internal Port "ha-da465bd8-6a" tag: 10 Interface "ha-da465bd8-6a" type: internal Port int-br-sahara Interface int-br-sahara type: patch options: {peer=phy-br-sahara} Port "tapb875eb8b-7a" tag: 3 Interface "tapb875eb8b-7a" type: internal Port "ha-45f6fa26-ec" tag: 8 Interface "ha-45f6fa26-ec" type: internal Port "ha-895752ed-6b" tag: 9 Interface "ha-895752ed-6b" type: internal Port "qr-0ef11b01-d8" tag: 25 Interface "qr-0ef11b01-d8" type: internal Port "qr-ae359120-fd" tag: 7 Interface "qr-ae359120-fd" type: internal Port "qg-f4ff7324-e2" tag: 12 Interface "qg-f4ff7324-e2" type: internal Port "tap0b7401cf-1f" tag: 7 Interface "tap0b7401cf-1f" type: internal Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port "qg-265ebbb0-63" tag: 12 Interface "qg-265ebbb0-63" type: internal Port br-int Interface br-int type: internal Port "qr-34b1985c-e5" tag: 17 Interface "qr-34b1985c-e5" type: internal Port "tap7363efaf-a7" tag: 25 Interface "tap7363efaf-a7" type: internal Bridge br-sahara Controller "tcp:127.0.0.1:6633" fail_mode: secure Port "p3p1.207" Interface "p3p1.207" Port br-sahara Interface br-sahara type: internal Port phy-br-sahara Interface phy-br-sahara type: patch options: {peer=int-br-sahara} Bridge br-ex Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port br-ex Interface br-ex type: internal Port "p3p1.3802" Interface "p3p1.3802" Port phy-br-ex Interface phy-br-ex type: patch options: {peer=int-br-ex} ovs_version: "2.9.0" And here's where things start to look suspicious: [root@neu-17-11-nc2 ~]# ovs-ofctl dump-flows br-sahara [root@neu-17-11-nc2 ~]# I would expect to see rules there that match (and strip) the vlan tags on the associated br-int ports. What's extra odd is that I see what looks like a matching rule on br-ex (the second rule matches dl_vlan=14): [root@neu-17-11-nc2 ~]# ovs-ofctl dump-flows br-ex cookie=0xb2514f67fe7d2fc1, duration=86985.002s, table=0, n_packets=2559880, n_bytes=251421650, priority=4,in_port="phy-br-ex",dl_vlan=12 actions=strip_vlan,NORMAL cookie=0x79c8c5bf077fd19e, duration=86984.952s, table=0, n_packets=47, n_bytes=2814, priority=4,in_port="phy-br-ex",dl_vlan=14 actions=strip_vlan,NORMAL cookie=0xb2514f67fe7d2fc1, duration=87001.704s, table=0, n_packets=220443, n_bytes=12048444, priority=2,in_port="phy-br-ex" actions=drop cookie=0xb2514f67fe7d2fc1, duration=87001.733s, table=0, n_packets=5441915, n_bytes=18036033429, priority=0 actions=NORMAL What's going on here?
When I create a new router and `set --external-gateway` to the `br-sahara` network, I see on nc1: [root@neu-19-11-nc1 ~]# tail /var/log/containers/neutron/openvswitch-agent.log [...] 2018-11-30 15:31:01.677 21339 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-08de5965-b30d-4a5f-8ea7-9567ec000130 - - - - -] Port 25c21655-e058-4d51-9e5d-7de455bda0b3 updated. Details: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '2ac07a0f-5428-449a-9d8f-e93ac680d2cb', 'segmentation_id': None, 'fixed_ips': [{'subnet_id': '8c5b00c5-b08c-47e7-8414-9251e1df5346', 'ip_address': '10.251.0.50'}], 'device_owner': u'network:router_gateway', 'physical_network': u'sahara', 'mac_address': 'fa:16:3e:d7:a5:ab', 'device': u'25c21655-e058-4d51-9e5d-7de455bda0b3', 'port_security_enabled': False, 'port_id': '25c21655-e058-4d51-9e5d-7de455bda0b3', 'network_type': u'flat', 'security_groups': []} 2018-11-30 15:31:01.677 21339 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-08de5965-b30d-4a5f-8ea7-9567ec000130 - - - - -] Assigning 15 as local vlan for net-id=2ac07a0f-5428-449a-9d8f-e93ac680d2cb 2018-11-30 15:31:01.689 21339 INFO neutron.agent.securitygroups_rpc [req-08de5965-b30d-4a5f-8ea7-9567ec000130 - - - - -] Preparing filters for devices set([u'25c21655-e058-4d51-9e5d-7de455bda0b3']) 2018-11-30 15:31:02.123 21339 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-08de5965-b30d-4a5f-8ea7-9567ec000130 - - - - -] Configuration for devices up [u'25c21655-e058-4d51-9e5d-7de455bda0b3'] and devices down [] completed. 2018-11-30 15:31:03.677 21339 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-08de5965-b30d-4a5f-8ea7-9567ec000130 - - - - -] Port 25c21655-e058-4d51-9e5d-7de455bda0b3 updated. Details: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '2ac07a0f-5428-449a-9d8f-e93ac680d2cb', 'segmentation_id': None, 'fixed_ips': [{'subnet_id': '8c5b00c5-b08c-47e7-8414-9251e1df5346', 'ip_address': '10.251.0.50'}], 'device_owner': u'network:router_gateway', 'physical_network': u'sahara', 'mac_address': 'fa:16:3e:d7:a5:ab', 'device': '25c21655-e058-4d51-9e5d-7de455bda0b3', 'port_security_enabled': False, 'port_id': '25c21655-e058-4d51-9e5d-7de455bda0b3', 'network_type': u'flat', 'security_groups': []} 2018-11-30 15:31:03.681 21339 INFO neutron.agent.securitygroups_rpc [req-08de5965-b30d-4a5f-8ea7-9567ec000130 - - - - -] Refresh firewall rules 2018-11-30 15:31:03.835 21339 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-08de5965-b30d-4a5f-8ea7-9567ec000130 - - - - -] Configuration for devices up ['25c21655-e058-4d51-9e5d-7de455bda0b3'] and devices down [] completed. [root@neu-19-11-nc1 ~]# ovs-ofctl dump-flows br-sahara cookie=0xeb7d6f5281b073d5, duration=100.215s, table=0, n_packets=12, n_bytes=988, priority=4,in_port="phy-br-sahara",dl_vlan=15 actions=strip_vlan,NORMAL And on nc2: [root@neu-17-11-nc2 neutron]# tail -f /var/log/containers/neutron/openvswitch-agent.log [...] 2018-11-30 15:31:01.585 719274 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-b7379065-19de-42b8-9be9-3891dd4a4d7b - - - - -] Port 25c21655-e058-4d51-9e5d-7de455bda0b3 updated. Details: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '2ac07a0f-5428-449a-9d8f-e93ac680d2cb', 'segmentation_id': None, 'fixed_ips': [{'subnet_id': '8c5b00c5-b08c-47e7-8414-9251e1df5346', 'ip_address': '10.251.0.50'}], 'device_owner': u'network:router_gateway', 'physical_network': u'sahara', 'mac_address': 'fa:16:3e:d7:a5:ab', 'device': u'25c21655-e058-4d51-9e5d-7de455bda0b3', 'port_security_enabled': False, 'port_id': '25c21655-e058-4d51-9e5d-7de455bda0b3', 'network_type': u'flat', 'security_groups': []} 2018-11-30 15:31:01.585 719274 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-b7379065-19de-42b8-9be9-3891dd4a4d7b - - - - -] Assigning 2 as local vlan for net-id=2ac07a0f-5428-449a-9d8f-e93ac680d2cb 2018-11-30 15:31:01.598 719274 INFO neutron.agent.securitygroups_rpc [req-b7379065-19de-42b8-9be9-3891dd4a4d7b - - - - -] Preparing filters for devices set([u'25c21655-e058-4d51-9e5d-7de455bda0b3']) 2018-11-30 15:31:01.699 719274 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-b7379065-19de-42b8-9be9-3891dd4a4d7b - - - - -] Configuration for devices up [u'25c21655-e058-4d51-9e5d-7de455bda0b3'] and devices down [] completed. 2018-11-30 15:31:03.597 719274 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-b7379065-19de-42b8-9be9-3891dd4a4d7b - - - - -] Port 25c21655-e058-4d51-9e5d-7de455bda0b3 updated. Details: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '2ac07a0f-5428-449a-9d8f-e93ac680d2cb', 'segmentation_id': None, 'fixed_ips': [{'subnet_id': '8c5b00c5-b08c-47e7-8414-9251e1df5346', 'ip_address': '10.251.0.50'}], 'device_owner': u'network:router_gateway', 'physical_network': u'sahara', 'mac_address': 'fa:16:3e:d7:a5:ab', 'device': '25c21655-e058-4d51-9e5d-7de455bda0b3', 'port_security_enabled': False, 'port_id': '25c21655-e058-4d51-9e5d-7de455bda0b3', 'network_type': u'flat', 'security_groups': []} 2018-11-30 15:31:03.603 719274 INFO neutron.agent.securitygroups_rpc [req-b7379065-19de-42b8-9be9-3891dd4a4d7b - - - - -] Refresh firewall rules 2018-11-30 15:31:03.707 719274 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-b7379065-19de-42b8-9be9-3891dd4a4d7b - - - - -] Configuration for devices up ['25c21655-e058-4d51-9e5d-7de455bda0b3'] and devices down [] completed. [root@neu-17-11-nc2 neutron]# ovs-ofctl dump-flows br-sahara [root@neu-17-11-nc2 neutron]# So the logs on both hosts seem identical, but the configuration of br-sahara is different (and in both cases, incorrect).
This difference bothers me a little: Bridge br-sahara Controller "tcp:127.0.0.1:6633" fail_mode: secure ...} Bridge br-ex Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure ... I believe it shows br-sahara isn't connected to the controller. Are there any debug messages in ovs-vswitchd.log about this? For example there will usually be something like this: 2018-12-05T16:40:18.030Z|00039|connmgr|INFO|br-ex: added primary controller "tcp:127.0.0.1:6633" 2018-12-05T16:40:18.030Z|00040|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connecting... 2018-12-05T16:40:18.035Z|00041|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connected
Brian, I see this over and over in ovs-switchd.log on one of the two controllers: 2018-12-05T19:05:07.361Z|745872|rconn|INFO|br-sahara<->tcp:127.0.0.1:6633: connected 2018-12-05T19:05:08.362Z|745876|rconn|INFO|br-sahara<->tcp:127.0.0.1:6633: connection closed by peer 2018-12-05T19:05:16.362Z|745877|rconn|INFO|br-sahara<->tcp:127.0.0.1:6633: connected 2018-12-05T19:05:17.363Z|745881|rconn|INFO|br-sahara<->tcp:127.0.0.1:6633: connection closed by peer 2018-12-05T19:05:25.361Z|745882|rconn|INFO|br-sahara<->tcp:127.0.0.1:6633: connected 2018-12-05T19:05:26.362Z|745886|rconn|INFO|br-sahara<->tcp:127.0.0.1:6633: connection closed by peer 2018-12-05T19:05:34.362Z|745887|rconn|INFO|br-sahara<->tcp:127.0.0.1:6633: connected 2018-12-05T19:05:35.362Z|745891|rconn|INFO|br-sahara<->tcp:127.0.0.1:6633: connection closed by peer
Created attachment 1511892 [details] the output of "grep -r sahara /var/log/containers/neutron" on nc1
On nc2 (the system on which br-sahara appears to be disconnected) there is no match for "sahara" in "grep -r sahara /var/log/containers/neutron". I've attached the output of the same command running on the other networker, nc1.
So I guess we'll need to figure out why br-sahara is disconnected. Getting access to the system would be great, and/or an sos-report, since it would be good to take a look at all the /etc/neutron files and the logs. For example, the l3-agent can support multiple external networks, but it could be configured wrong, see https://review.openstack.org/#/c/567369/ for more info on that - a patch to remove some deprecated external network settings from the agent.
It looks like, as I suspected, this is actually a dupe of #1654840. *** This bug has been marked as a duplicate of bug 1654840 ***