1022578 – quantum-ovs-cleanup delete the port that nova compute has plugged into an tap interface

Bug 1022578 - quantum-ovs-cleanup delete the port that nova compute has plugged into an tap interface

Summary: quantum-ovs-cleanup delete the port that nova compute has plugged into an tap...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-quantum
Sub Component:
Version:	3.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	z4
Target Release:	3.0
Assignee:	Solly Ross
QA Contact:	Ofer Blaut
Docs Contact:
URL:
Whiteboard:	network
Depends On:	1010941
Blocks:
TreeView+	depends on / blocked

Reported:	2013-10-23 14:53 UTC by Scott Lewis
Modified:	2019-09-09 16:05 UTC (History)
CC List:	15 users (show)
Fixed In Version:	openstack-quantum-2013.1.4-3.el6ost
Doc Type:	Bug Fix
Doc Text:	Cause: The ovs cleanup script would exit before its operations finished. Consequence: When restarting services, it was possible that OVS cleanup would remove devices needed by newly restarted services. Fix: Ensure that the ovs cleanup script does not exit before completion. Result: OVS cleanup does not delete devices that are created after it is run.
Clone Of:	1010941
Environment:
Last Closed:	2014-01-30 19:49:50 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2014:0110	0	normal	SHIPPED_LIVE	Red Hat Enterprise Linux OpenStack Platform 3 bug fix advisory	2014-01-31 00:48:26 UTC

Comment 3 Ofer Blaut 2013-10-31 14:08:30 UTC

I have followed the steps but VM has no traffic , 
OVS 
What should happen 

cleanup-logs
2013-10-31 09:11:17     INFO [quantum.common.config] Logging enabled!
2013-10-31 09:11:17     INFO [quantum.agent.ovs_cleanup_util] Cleaning br-int
2013-10-31 09:11:18     INFO [quantum.agent.ovs_cleanup_util] OVS cleanup completed successfully
2013-10-31 12:40:42     INFO [quantum.common.config] Logging enabled!
2013-10-31 12:40:43     INFO [quantum.agent.ovs_cleanup_util] Cleaning br-int
2013-10-31 12:40:45     INFO [quantum.agent.ovs_cleanup_util] Delete qvo2ed438a8-a5
2013-10-31 12:40:46     INFO [quantum.agent.ovs_cleanup_util] Delete qvo4f81529e-a9
2013-10-31 12:40:47     INFO [quantum.agent.ovs_cleanup_util] Delete qvoc2cdc2cc-2c
2013-10-31 12:40:47     INFO [quantum.agent.ovs_cleanup_util] OVS cleanup completed successfully
2013-10-31 12:52:36     INFO [quantum.common.config] Logging enabled!
2013-10-31 12:52:37     INFO [quantum.agent.ovs_cleanup_util] Cleaning br-int
2013-10-31 12:52:37     INFO [quantum.agent.ovs_cleanup_util] OVS cleanup completed successfully
~                                                                                                    
OVS-logs

2013-10-31 11:03:14     INFO [quantum.openstack.common.rpc.impl_qpid] Connected to AMQP server on 10.35.160.17:5672
2013-10-31 11:03:15     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Port c2cdc2cc-2c42-43f5-9f3f-05702625746d added
2013-10-31 11:03:15     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Port c2cdc2cc-2c42-43f5-9f3f-05702625746d updated. Details: {u'admin_state_up': True, u'network_id': u'b7f821d3-b437-400e-b294-e43aa1330184', u'segmentation_id': 201, u'physical_network': u'inter-vlan', u'device': u'c2cdc2cc-2c42-43f5-9f3f-05702625746d', u'port_id': u'c2cdc2cc-2c42-43f5-9f3f-05702625746d', u'network_type': u'vlan'}
2013-10-31 11:03:15     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Assigning 1 as local vlan for net-id=b7f821d3-b437-400e-b294-e43aa1330184
2013-10-31 11:03:16     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Port 2ed438a8-a585-40d8-8df2-f89ccd610858 added
2013-10-31 11:03:16     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Port 2ed438a8-a585-40d8-8df2-f89ccd610858 updated. Details: {u'admin_state_up': True, u'network_id': u'a5fb3fe2-a1bb-439b-95c2-a69b844cc185', u'segmentation_id': 202, u'physical_network': u'inter-vlan', u'device': u'2ed438a8-a585-40d8-8df2-f89ccd610858', u'port_id': u'2ed438a8-a585-40d8-8df2-f89ccd610858', u'network_type': u'vlan'}
2013-10-31 11:03:16     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Assigning 2 as local vlan for net-id=a5fb3fe2-a1bb-439b-95c2-a69b844cc185
2013-10-31 11:03:16     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Port 4f81529e-a9a9-467a-b6a3-c18b8d86a7f8 added
2013-10-31 11:03:17     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Port 4f81529e-a9a9-467a-b6a3-c18b8d86a7f8 updated. Details: {u'admin_state_up': True, u'network_id': u'b7f821d3-b437-400e-b294-e43aa1330184', u'segmentation_id': 201, u'physical_network': u'inter-vlan', u'device': u'4f81529e-a9a9-467a-b6a3-c18b8d86a7f8', u'port_id': u'4f81529e-a9a9-467a-b6a3-c18b8d86a7f8', u'network_type': u'vlan'}
2013-10-31 11:40:18     INFO [quantum.agent.securitygroups_rpc] Security group member updated [u'9e9f2c59-b3bf-4d1e-be0b-135f9f944e4e']
2013-10-31 11:40:18     INFO [quantum.agent.securitygroups_rpc] Refresh firewall rules
2013-10-31 11:40:18     INFO [quantum.agent.securitygroups_rpc] Provider rule updated
2013-10-31 11:40:18     INFO [quantum.agent.securitygroups_rpc] Refresh firewall rules
2013-10-31 11:52:00     INFO [quantum.agent.securitygroups_rpc] Security group member updated [u'9e9f2c59-b3bf-4d1e-be0b-135f9f944e4e']
2013-10-31 11:52:00     INFO [quantum.agent.securitygroups_rpc] Refresh firewall rules
2013-10-31 11:52:00     INFO [quantum.agent.securitygroups_rpc] Security group member updated [u'9e9f2c59-b3bf-4d1e-be0b-135f9f944e4e']
2013-10-31 11:52:00     INFO [quantum.agent.securitygroups_rpc] Refresh firewall rules
2013-10-31 12:31:35     INFO [quantum.agent.securitygroups_rpc] Security group rule updated [u'9e9f2c59-b3bf-4d1e-be0b-135f9f944e4e']
2013-10-31 12:31:35     INFO [quantum.agent.securitygroups_rpc] Refresh firewall rules
2013-10-31 12:41:07     INFO [quantum.common.config] Logging enabled!
2013-10-31 12:41:08     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Mapping physical network inter-vlan to bridge br-eth3
2013-10-31 12:41:09     INFO [quantum.openstack.common.rpc.impl_qpid] Connected to AMQP server on 10.35.160.17:5672
2013-10-31 12:41:10     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Agent initialized successfully, now running...
2013-10-31 12:41:10     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Agent out of sync with plugin!
2013-10-31 12:41:10     INFO [quantum.openstack.common.rpc.impl_qpid] Connected to AMQP server on 10.35.160.17:5672
2013-10-31 12:54:30     INFO [quantum.common.config] Logging enabled!
2013-10-31 12:54:30     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Mapping physical network inter-vlan to bridge br-eth3
2013-10-31 12:54:32     INFO [quantum.openstack.common.rpc.impl_qpid] Connected to AMQP server on 10.35.160.17:5672
2013-10-31 12:54:32     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Agent initialized successfully, now running...
2013-10-31 12:54:32     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Agent out of sync with plugin!
2013-10-31 12:54:32     INFO [quantum.openstack.common.rpc.impl_qpid] Connected to AMQP server on 10.35.160.17:5672

Comment 4 Ofer Blaut 2013-11-04 07:59:31 UTC

Can't validate the bug , it is not working

Comment 9 Terry Wilson 2013-11-05 04:13:02 UTC

Ofer: the steps to reproduce are how to produce the error artificially, in a way that would never happen normally. Testing shouldn't be by the above steps to reproduce, but instead should just verify that normal startup doesn't fail. The fix for this was to make sure that the scripts, when running with their normal priorities, don't occasionally fail due to the race condition that existed because daemon launched the OVS cleanup utility in a non-blocking way which could break nova if it ended up running afterward.

As it was a race condition, the only way to really try it out would be to go through lots of reboots. This shouldn't have been removed from z-stream in my opinion.

Comment 10 Ofer Blaut 2013-11-05 06:03:03 UTC

Hi Terry, the normal start up works, i just followed the reproduce steps as required . 

As i mentioned in my comments from "comment from Ofer Blaut on 2013-10-23" i was not sure about the steps.

Since our nova.conf default is "resume_guests_state_on_host_boot=False"

I will change it to True and reboot the hosts to check it out .

Comment 11 Ofer Blaut 2013-11-05 13:01:42 UTC

I have tried few times with start_guests_on_host_boot=True

Can't get the VMS UP after boot

Comment 12 Ofer Blaut 2013-11-19 18:38:14 UTC

I have just checked it, The reproduction steps are

1. stop ovs-agent
2. stop nova-compute
3. run cleanup
4. start ovs-agent
5. start nova-compute

Only after restarting nova compute the interfaces are back

Comment 13 Ofer Blaut 2014-01-22 15:12:25 UTC

already tested ( see comment 12 )

Comment 17 errata-xmlrpc 2014-01-30 19:49:50 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0110.html

Note You need to log in before you can comment on or make changes to this bug.