Bug 1257567

If a restart of neutron-openvswitch-agent-clone fixes it, then the mostly likely candidate is a missing ordering constraint is allowing neutron-openvswitch-agent-clone to start too early.

I'll investigate if this is the case.

Unfortunately there is are no sos or crm reports present, so I cannot investigate further.

It looks like the required constraints are present.

We can make the start of neutron-openvswitch-agent-clone conditional on another component if needed, but someone would need to tell us which component.

Personally I have no knowledge of what these tags are.

Created attachment 1077111 [details]
tiny tool to merge the neutron logs

Something I built from scratch to do it, just use as:

python os_merge_logs.py */var/log/neutron/*.log > synclogs

It will sync all logs from different sources, while adding a new column after date which specifies the source file.

After looking at the logs, all the tag=4095 belong to ports which neutron server reported as binding failed.

I'd like to see the logs of failure (and then) the recovery (when neutron-openvswitch-agent gets restarted) of ova-agents and server in DEBUG.

Is that still available for the current logs?

@andrew, it seems like neutron-server is in place before the openvswitch-agents start, that's all we need (database and rabbitmq will be there if neutron-server is already up)

Ok, I think I got to the culprit of the issue.

1) neutron-servers and agents are down by being set to standby

some time, T>agent_down_time

2) cluster is took out of standby, neutron-servers and agents are started again, first neutron-server, then agents

3) neutron servers consider the agents to be down (last heartbeat > agent_down_time), when checking ports in the database, reset them as binding failed...

4) agents start, try to fetch info from the ports, as those are in status binding failed... they get no info from the server (or the interpretation is wrong "Device xxx not defined on plugin") 

5) agents mark ports as dead vlan (4095), as the port "does not exist" on the server


Analysis over the logs:
  

-- SRV2 tryies to bind port on dead agent (: same happening for SRV1 and 3 that's omitted---

2015-08-25 09:37:11.687000 [SRV2] 16522 DEBUG neutron.plugins.ml2.managers [req-fd4c17c2-0acc-4bb6-a652-91359eda496b None] Attempting to bind port 6adcffbf-09d5-4a85-9339-9d6beb2bf82c on host neutron-n-0 for vnic_type normal with profile  bind_port /usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py:574

2015-08-25 09:37:11.687000 [SRV2] 16522 DEBUG neutron.plugins.ml2.drivers.mech_agent [req-fd4c17c2-0acc-4bb6-a652-91359eda496b None] Attempting to bind port 6adcffbf-09d5-4a85-9339-9d6beb2bf82c on network 2c52ca73-b11f-426f-9b78-185ef15809e3 bind_port /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/mech_agent.py:57

2015-08-25 09:37:11.689000 [SRV2] 16522 DEBUG neutron.plugins.ml2.drivers.mech_agent [req-fd4c17c2-0acc-4bb6-a652-91359eda496b None] Checking agent: {'binary': u'neutron-openvswitch-agent', 'description': None, 'admin_state_up': True, 'heartbeat_timestamp': datetime.datetime(2015, 8, 25, 1, 35, 5), 'alive': False, 'id': u'196101f4-9680-43bb-8310-9fca49cd4930', 'topic': u'N/A', 'host': u'neutron-n-0', 'agent_type': u'Open vSwitch agent', 'started_at': datetime.datetime(2015, 8, 22, 6, 55, 59), 'created_at': datetime.datetime(2015, 8, 12, 10, 23, 16), 'configurations': {u'arp_responder_enabled': False, u'tunneling_ip': u'', u'devices': 28, u'l2_population': False, u'tunnel_types': [], u'enable_distributed_routing': False, u'bridge_mappings': {u'physnet-external': u'br-ex', u'physnet-tenants': u'br-enp7s0'}}} bind_port /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/mech_agent.py:65

2015-08-25 09:37:11.690000 [SRV2] 16522 WARNING neutron.plugins.ml2.drivers.mech_agent [req-fd4c17c2-0acc-4bb6-a652-91359eda496b None] Attempting to bind with dead agent: {'binary': u'neutron-openvswitch-agent', 'description': None, 'admin_state_up': True, 'heartbeat_timestamp': datetime.datetime(2015, 8, 25, 1, 35, 5), 'alive': False, 'id': u'196101f4-9680-43bb-8310-9fca49cd4930', 'topic': u'N/A', 'host': u'neutron-n-0', 'agent_type': u'Open vSwitch agent', 'started_at': datetime.datetime(2015, 8, 22, 6, 55, 59), 'created_at': datetime.datetime(2015, 8, 12, 10, 23,16), 'configurations': {u'arp_responder_enabled': False, u'tunneling_ip': u'', u'devices': 28, u'l2_population': False, u'tunnel_types': [], u'enable_distributed_routing': False, u'bridge_mappings': {u'physnet-external': u'br-ex', u'physnet-tenants': u'br-enp7s0'}}}

2015-08-25 09:37:11.690000 [SRV2] 16522 WARNING neutron.plugins.ml2.managers [req-fd4c17c2-0acc-4bb6-a652-91359eda496b None] Failed to bind port 6adcffbf-09d5-4a85-9339-9d6beb2bf82c on host neutron-n-0

2015-08-25 09:37:11.719000 [SRV2] 16522 WARNING neutron.plugins.ml2.plugin [req-fd4c17c2-0acc-4bb6-a652-91359eda496b None] In _notify_port_updated(), no bound segment for port 6adcffbf-09d5-4a85-9339-9d6beb2bf82c on network 2c52ca73-b11f-426f-9b78-185ef15809e3


-- OVS agent 1 requests info for ports, including 6adcffbf-09d5-4a85-9339-9d6beb2bf82c --


2015-08-25 09:37:16.043000 [OVS1] 12613 DEBUG neutron.common.rpc [req-588c942a-6526-464f-a447-782a5e2d436a None] neutron.plugins.openvswitch.agent.ovs_neutron_agent.OVSPluginApi method call called with arguments (<neutron.context.ContextBase object at 0x326d510>, {'args': {'host': 'neutron-n-2', 'agent_id': 'ovs-agent-neutron-n-2', 'devices': set([u'6adcffbf-09d5-4a85-9339-9d6beb2bf82c', u'2cdba01d-e48a-46a6-82c0-bbbfd8e2b490', u'fa58d644-8162-410b-8b45-747ae8813a61', u'5d5ea109-4807-4df3-bef4-b5d89c3ffebc', u'743ccaa6-7ed9-4195-aabd-3d55006338e1', u'dc662767-61a5-4807-b2ed-a7c76b541fd6', u'b5b271a1-d8b7-4b55-86fd-6316523878a5', u'b2d04f15-f2e2-429c-b948-00f88e1b9cf6', u'34b826df-9787-443c-9bef-084374827a85', u'c12e6e06-ff6a-44dc-b75f-78ec55dd3dd3', u'79b33879-3232-4b3a-a27c-c0a79da10379', u'499d3db7-9737-43a4-bcbf-923f9df5196e', u'55a19f5e-efff-4342-a82a-70ddb33fbefb', u'44dca3a9-44d4-418f-9d60-99dfe37afdc6'])}, 'namespace': None, 'method': 'get_devices_details_list'}) {'version': '1.3'} wrapper /usr/lib/python2.7/site-packages/neutron/common/log.py:33


---- SRV1 notes the port is in binding_failed state ----

2015-08-25 09:37:16.048000 [SRV1] 15065 DEBUG neutron.plugins.ml2.rpc [req-588c942a-6526-464f-a447-782a5e2d436a None] Device 6adcffbf-09d5-4a85-9339-9d6beb2bf82c details requested by agent ovs-agent-neutron-n-2 with host neutron-n-2 get_device_details /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:60

2015-08-25 09:37:16.174000 [SRV1] 15065 WARNING neutron.plugins.ml2.rpc [req-588c942a-6526-464f-a447-782a5e2d436a None] Device 6adcffbf-09d5-4a85-9339-9d6beb2bf82c requested by agent ovs-agent-neutron-n-2 on network 2c52ca73-b11f-426f-9b78-185ef15809e3 not bound, vif_type: binding_failed


2015-08-25 09:37:16.807000 [OVS1] 12613 WARNING neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-588c942a-6526-464f-a447-782a5e2d436a None] Device 6adcffbf-09d5-4a85-9339-9d6beb2bf82c not defined on plugin

--- OVS1 tags it out ---

2015-08-25 09:37:16.900000 [OVS1] 12613 DEBUG neutron.agent.linux.utils [req-588c942a-6526-464f-a447-782a5e2d436a None] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', 'set', 'Port', 'qr-6adcffbf-09', 'tag=4095'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:48

I will try to reproduce this in a simpler environment (AllInOne OSP6 and AllInOne OSP7) to se if this is fixed in later codebases.

@amuller, do you remember anything about this? I have the feeling it was fixed, at least IMO, as agents come back online, the ports should be rebound, and not left in binding failed state.

I'm trying to reproduce with openstack-neutron-2014.2.3-23.el7ost.noarch with no success, and I see the report saying openstack-neutron-2014.2.3-19.el7ost.noarch anyway, I don't see any related changes from -19 to -24

I will try to reproduce.

Could we get a:

# pcs constraint list   

from customer?

(just to verify l3 has a constraint on ovs)

I mean, I will try to reproduce it again, I have more insight on the possible reason thanks to Assaf.

Ok, just managed to reproduce it on osp6, trying osp7 to check if there's any difference.

needinfo on the # pcs constraint list

it could be useful to better understand why is it happening.

Ok, I have managed to reproduce this on OSP6 and OSP7, and I also found that what fixes the issue is restarting the neutron-l3-agent (l3-agent is also restarted by the dependency on neutron-openvswitch-agent-clone  via pcs restart) 

Also it only happens on the qr-* ports of l3-ha routers.

Now I'm investigating how to fix this.

@abeekhof, is there any possibility to insert a delay in the dependency chain, in this case, on the current codebase, we don't want the l3-agent to be started at par with neutron-openvswitch-agent (or very quickly after), we just need neutron-openvswitch-agent to be able to send the first heartbeat to neutron-server before l3-agent starts.

I guess we should also revise the readiness notification from the neutron-openvswitch-agent to make sure it's not sent until the first heartbeat has been sent to neutron-server.

  /usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py(130)_do_dispatch()
-> result = func(ctxt, **new_args)
  /usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py(86)sync_routers()
-> self._ensure_host_set_on_ports(context, host, routers)
  /usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py(108)_ensure_host_set_on_ports()
-> router['id'])
  /usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py(128)_ensure_host_set_on_port()
-> {'port': {portbindings.HOST_ID: host}})
  /usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py(1142)update_port()
-> need_notify=need_port_update_notify)
  /usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py(293)_bind_port_if_needed()
-> bind_context = self._bind_port(context)
  /usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py(331)_bind_port()
-> self.mechanism_manager.bind_port(new_context)
> /usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py(617)bind_port()
-> LOG.error(_LE("Failed to bind port %(port)s on host %(host)s"),

(In reply to Miguel Angel Ajo from comment #29)
> Ok, I have managed to reproduce this on OSP6 and OSP7, and I also found that
> what fixes the issue is restarting the neutron-l3-agent (l3-agent is also
> restarted by the dependency on neutron-openvswitch-agent-clone  via pcs
> restart) 
> 
> Also it only happens on the qr-* ports of l3-ha routers.
> 
> Now I'm investigating how to fix this.
> 
> @abeekhof, is there any possibility to insert a delay in the dependency
> chain, in this case, on the current codebase, we don't want the l3-agent to
> be started at par with neutron-openvswitch-agent (or very quickly after), we
> just need neutron-openvswitch-agent to be able to send the first heartbeat
> to neutron-server before l3-agent starts.
> 
> I guess we should also revise the readiness notification from the
> neutron-openvswitch-agent to make sure it's not sent until the first
> heartbeat has been sent to neutron-server.

Bingo :)

The use of a delay usually indicates there is a more fundamental problem (the other resource is saying "I'm started" too early).

Better to fix the agent/service than to add a delay:
- the delay will always be larger than the real amount of time needed == wasted
- if more than one resource depends on it, then you need multiple delay resources too


But if adding a delay helps get concentrate on the more pressing issue, go for it.
Ideally we'd have the "proper" fix (the revised neutron-openvswitch-agent readiness notification) before this went to customers though.

Thanks, I totally agree.

@abeekhof, how could we add a delay on the ordering constraint (already existing) from the neutron-l3-agent (depending on) the neutron-openvswitch-agent ? I'd like to share that manual workaround to customer for test while I work on fixing the openvswitch-agent.

Ok, for what I see, only neutron-server provides systems readiness notification: 

$ grep Type=notify *.service
neutron-server.service:Type=notify

and: 

neutron/wsgi.py:from neutron.openstack.common import systemd
neutron/wsgi.py:            systemd.notify_once()


IMHO we should incorporate this to all the agents after they send the first heartbeat to neutron-server declaring themselves as UP. That introduces a hard requisite that neutron-server must be up, which could be too much for people which don't care about this. So probably this could be a configurable option for agents.

I will discuss this U/S.

oslo.service already notifies systemd. We may want to influence the time of the notification though (not sure the library allows for it).

Also consider that systemd units have timeout that could result in services down if neutron-server does not respond quick enough. Not sure what it would mean for us. Could we get into some kind of unrecoverable failure mode where services consistently fail to start and then restarted by external service monitoring (pacemaker?) and repeat the loop.

The simplest way might be to set start-delay=60s (or whatever number makes sense) for the start operation of the l3 agent.  I forget what the resource name of the agent is, but assuming "l3-agent", then command would be:

  pcs resource update l3-agent op start start-delay=60s

(In reply to Ihar Hrachyshka from comment #35)
> oslo.service already notifies systemd. We may want to influence the time of
> the notification though (not sure the library allows for it).

It doesn't seem to be an issue for the ova-agent yet because it's not depending on the oslo service framework, we should suggest the feature to oslo service, by the way..


> Also consider that systemd units have timeout that could result in services
> down if neutron-server does not respond quick enough. Not sure what it would
> mean for us. Could we get into some kind of unrecoverable failure mode where
> services consistently fail to start and then restarted by external service
> monitoring (pacemaker?) and repeat the loop.

Interesting point.

We have a dependency on neutron-server, so, it should be partly ok to start the agent anytime, pacemaker will make sure neutron-server is available first. If agent has an issue contacting the server is better to move it into error state: could be due to connectivity issues, that's ok, could be due to excessive neutron-server load, in that case, we will also prevent other agents from starting after ova-agent which is a good thing (since we limit the server load).

I'm not sure about what pacemaker would do in such case - ovs agent fails to start- ( @abeekhof? ), we have two scenarios: neutron A/P configuration (as it was deployed for OSP6), and A/A neutron (with ha-routers)

Jaison, could we try if this workaround works?

pcs resource update neutron-l3-agent op start start-delay=30s

We're working on a more definitive solution to perfectly synchronize both, but that's still under discussion. I will fill an U/S bug for this btw.

The aforementioned definitive solution may be impacted by the issue indicated in this bug report:

https://bugzilla.redhat.com/show_bug.cgi?id=1111747  Pacemaker doesn't wait for systemd resource start. 

Deployers should ensure that the installed version of pacemaker is of pacemaker-1.1.12-4 or newer when evaluating the fix for this issue.

Sorry, I didn't see the question for me.

(In reply to Miguel Angel Ajo from comment #37)
> > Also consider that systemd units have timeout that could result in services
> > down if neutron-server does not respond quick enough. Not sure what it would
> > mean for us. Could we get into some kind of unrecoverable failure mode where
> > services consistently fail to start and then restarted by external service
> > monitoring (pacemaker?) and repeat the loop.
> 
> Interesting point.
> 
> We have a dependency on neutron-server, so, it should be partly ok to start
> the agent anytime, pacemaker will make sure neutron-server is available
> first. If agent has an issue contacting the server is better to move it into
> error state: could be due to connectivity issues, that's ok, could be due to
> excessive neutron-server load, in that case, we will also prevent other
> agents from starting after ova-agent which is a good thing (since we limit
> the server load).
> 
> I'm not sure about what pacemaker would do in such case - ovs agent fails to
> start- ( @abeekhof? ), we have two scenarios: neutron A/P configuration (as
> it was deployed for OSP6), and A/A neutron (with ha-routers)

For A/P, we'll try starting it on one of the backup nodes.
Every time it fails on a node, we exclude it and move on to the next until there is nowhere left to try.

For A/A, its cloned so its already running everywhere that it can.
So we simply leave it stopped on the current node.

In both cases one of two things needs to happen. Either the admin needs to run 'pcs resource cleanup neutron-server' or failure-timeout needs to have been configured for neutron-server.

At that point we'll have another try at starting it again.

I have i believe the same error on OSP6, been trying to configure a VXLAN on with ML2 plugin, the provider network flat network seems to work fine but the internal VXLAN lan stays in down state.

==> /var/log/neutron/server.log <==
2016-03-31 21:40:18.255 10590 WARNING neutron.plugins.ml2.managers [req-c67a68ee-070c-4ec4-9fd2-5ccff2ad18e1 None] Failed to bind port 762905a5-51e1-4cbd-ab9e-c7a1d4be6b09 on host node1.jamesnet.ca
2016-03-31 21:40:18.282 10590 WARNING neutron.plugins.ml2.plugin [req-c67a68ee-070c-4ec4-9fd2-5ccff2ad18e1 None] In _notify_port_updated(), no bound segment for port 762905a5-51e1-4cbd-ab9e-c7a1d4be6b09 on network c2ec976e-02ad-43db-9f3b-79a538a5140f
2016-03-31 21:40:20.585 10928 WARNING neutron.plugins.ml2.rpc [req-251b195c-c7f4-45a5-b565-930ad28d930a None] Device 762905a5-51e1-4cbd-ab9e-c7a1d4be6b09 requested by agent ovs-agent-node1.jamesnet.ca on network c2ec976e-02ad-43db-9f3b-79a538a5140f not bound, vif_type: binding_failed



==> /var/log/openvswitch/ovs-vswitchd.log <==
2016-03-31T21:40:19.526Z|00058|bridge|INFO|bridge br-int: added interface qr-762905a5-51 on port 6
2016-03-31T21:40:19.670Z|00059|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qr-762905a5-51 device failed: No such device
2016-03-31T21:40:19.674Z|00060|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on qr-762905a5-51 device failed: No such device
2016-03-31T21:40:20.849Z|00061|netdev_linux|WARN|qr-762905a5-51: removing policing failed: No such device
2016-03-31T21:40:20.916Z|00062|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 adds)


==> /var/log/neutron/openvswitch-agent.log <==
2016-03-31 21:40:20.722 11357 WARNING neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-251b195c-c7f4-45a5-b565-930ad28d930a None] Device 762905a5-51e1-4cbd-ab9e-c7a1d4be6b09 not defined on plugin


I then try to launch a VM into this internal VXLAN network and get this error

2016-03-31 21:54:58.295 10506 ERROR nova.compute.manager [-] [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c] Instance failed to spawn
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c] Traceback (most recent call last):
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2306, in _build_resources
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c]     yield resources
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2176, in _build_and_run_instance
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c]     block_device_info=block_device_info)
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2797, in spawn
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c]     write_to_disk=True)
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4491, in _get_guest_xml
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c]     context)
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4352, in _get_guest_config
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c]     flavor, virt_type)
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 352, in get_config
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c]     _("Unexpected vif_type=%s") % vif_type)
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c] NovaException: Unexpected vif_type=binding_failed
2016-03-31 21:54:58.295 10506 TRACE nova.compute.manager [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c] 
2016-03-31 21:54:58.296 10506 AUDIT nova.compute.manager [req-86de0a96-a749-4a14-910f-0c60d66aae35 None] [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c] Terminating instance
2016-03-31 21:54:58.301 10506 WARNING nova.virt.libvirt.driver [-] [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c] During wait destroy, instance disappeared.
2016-03-31 21:54:58.411 10506 INFO nova.virt.libvirt.driver [-] [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c] Deleting instance files /var/lib/nova/instances/5dd322d6-9947-4e3c-9bd2-e102a6dede9c_del
2016-03-31 21:54:58.412 10506 INFO nova.virt.libvirt.driver [-] [instance: 5dd322d6-9947-4e3c-9bd2-e102a6dede9c] Deletion of /var/lib/nova/instances/5dd322d6-9947-4e3c-9bd2-e102a6dede9c_del complete
2016-03-31 21:54:58.510 10506 INFO nova.scheduler.client.report [-] Compute_service record updated for ('node1.jamesnet.ca', 'node1.jamesnet.ca')

I don't really understand it as the ports appear to be there according to ovs-vsctl show , i realize they are considered dead VLAN 4095 but why?


    Bridge br-int
        fail_mode: secure
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "qr-ff8f11f7-03"
            tag: 4095
            Interface "qr-ff8f11f7-03"
                type: internal
        Port "qr-762905a5-51"
            tag: 4095
            Interface "qr-762905a5-51"
                type: internal
        Port "tapb1906a9a-89"
            tag: 4095
            Interface "tapb1906a9a-89"
                type: internal
        Port "qvoe08577b8-3b"
            tag: 1
            Interface "qvoe08577b8-3b"
        Port br-int
            Interface br-int
                type: internal


Packages
openstack-neutron-common-2014.2.3-33.el7ost.noarch
openstack-neutron-2014.2.3-33.el7ost.noarch
openstack-neutron-openvswitch-2014.2.3-33.el7ost.noarch
python-neutron-2014.2.3-33.el7ost.noarch
openstack-neutron-ml2-2014.2.3-33.el7ost.noarch
python-neutronclient-2.3.9-2.el7ost.noarch

I am not using pacemaker in my setup

Apart from the port rebinding issue mentioned in comment 0, which was fixed and backported to OSP 8, we've also identified another issue that can cause this bug in certain timings [1]. Anil and Ann are working on a patch for it. We'll have to see after the patch is merged but it may be possible to backport it to OSP 8, but not 6.

[1] https://bugs.launchpad.net/neutron/+bug/1597461

[1] Can't be backported to mitaka and liberty. Please see comments in u/s backport [2] 

[1] https://bugs.launchpad.net/neutron/+bug/1597461
[2] https://review.openstack.org/#/c/364407/1

Patch[1] available in OSP7 in build openstack-neutron-2015.1.4-22.el7ost
Upgrading to RHOS10 will be always better. 

https://review.openstack.org/#/c/364407/

Can we close this bug?

(In reply to anil venkata from comment #63)
> Can we close this bug?

Yes . 
Customer is okay with rhos10 having the fix.
Thanks for the fix in rhos10 .