Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 1022578 - quantum-ovs-cleanup delete the port that nova compute has plugged into an tap interface
quantum-ovs-cleanup delete the port that nova compute has plugged into an tap...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-quantum (Show other bugs)
3.0
Unspecified Unspecified
unspecified Severity unspecified
: z4
: 3.0
Assigned To: Solly Ross
Ofer Blaut
network
: ZStream
Depends On: 1010941
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-23 10:53 EDT by Scott Lewis
Modified: 2016-04-26 11:52 EDT (History)
16 users (show)

See Also:
Fixed In Version: openstack-quantum-2013.1.4-3.el6ost
Doc Type: Bug Fix
Doc Text:
Cause: The ovs cleanup script would exit before its operations finished. Consequence: When restarting services, it was possible that OVS cleanup would remove devices needed by newly restarted services. Fix: Ensure that the ovs cleanup script does not exit before completion. Result: OVS cleanup does not delete devices that are created after it is run.
Story Points: ---
Clone Of: 1010941
Environment:
Last Closed: 2014-01-30 14:49:50 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:0110 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform 3 bug fix advisory 2014-01-30 19:48:26 EST

  None (edit)
Comment 3 Ofer Blaut 2013-10-31 10:08:30 EDT
I have followed the steps but VM has no traffic , 
OVS 
What should happen 

cleanup-logs
2013-10-31 09:11:17     INFO [quantum.common.config] Logging enabled!
2013-10-31 09:11:17     INFO [quantum.agent.ovs_cleanup_util] Cleaning br-int
2013-10-31 09:11:18     INFO [quantum.agent.ovs_cleanup_util] OVS cleanup completed successfully
2013-10-31 12:40:42     INFO [quantum.common.config] Logging enabled!
2013-10-31 12:40:43     INFO [quantum.agent.ovs_cleanup_util] Cleaning br-int
2013-10-31 12:40:45     INFO [quantum.agent.ovs_cleanup_util] Delete qvo2ed438a8-a5
2013-10-31 12:40:46     INFO [quantum.agent.ovs_cleanup_util] Delete qvo4f81529e-a9
2013-10-31 12:40:47     INFO [quantum.agent.ovs_cleanup_util] Delete qvoc2cdc2cc-2c
2013-10-31 12:40:47     INFO [quantum.agent.ovs_cleanup_util] OVS cleanup completed successfully
2013-10-31 12:52:36     INFO [quantum.common.config] Logging enabled!
2013-10-31 12:52:37     INFO [quantum.agent.ovs_cleanup_util] Cleaning br-int
2013-10-31 12:52:37     INFO [quantum.agent.ovs_cleanup_util] OVS cleanup completed successfully
~                                                                                                    
OVS-logs

2013-10-31 11:03:14     INFO [quantum.openstack.common.rpc.impl_qpid] Connected to AMQP server on 10.35.160.17:5672
2013-10-31 11:03:15     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Port c2cdc2cc-2c42-43f5-9f3f-05702625746d added
2013-10-31 11:03:15     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Port c2cdc2cc-2c42-43f5-9f3f-05702625746d updated. Details: {u'admin_state_up': True, u'network_id': u'b7f821d3-b437-400e-b294-e43aa1330184', u'segmentation_id': 201, u'physical_network': u'inter-vlan', u'device': u'c2cdc2cc-2c42-43f5-9f3f-05702625746d', u'port_id': u'c2cdc2cc-2c42-43f5-9f3f-05702625746d', u'network_type': u'vlan'}
2013-10-31 11:03:15     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Assigning 1 as local vlan for net-id=b7f821d3-b437-400e-b294-e43aa1330184
2013-10-31 11:03:16     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Port 2ed438a8-a585-40d8-8df2-f89ccd610858 added
2013-10-31 11:03:16     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Port 2ed438a8-a585-40d8-8df2-f89ccd610858 updated. Details: {u'admin_state_up': True, u'network_id': u'a5fb3fe2-a1bb-439b-95c2-a69b844cc185', u'segmentation_id': 202, u'physical_network': u'inter-vlan', u'device': u'2ed438a8-a585-40d8-8df2-f89ccd610858', u'port_id': u'2ed438a8-a585-40d8-8df2-f89ccd610858', u'network_type': u'vlan'}
2013-10-31 11:03:16     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Assigning 2 as local vlan for net-id=a5fb3fe2-a1bb-439b-95c2-a69b844cc185
2013-10-31 11:03:16     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Port 4f81529e-a9a9-467a-b6a3-c18b8d86a7f8 added
2013-10-31 11:03:17     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Port 4f81529e-a9a9-467a-b6a3-c18b8d86a7f8 updated. Details: {u'admin_state_up': True, u'network_id': u'b7f821d3-b437-400e-b294-e43aa1330184', u'segmentation_id': 201, u'physical_network': u'inter-vlan', u'device': u'4f81529e-a9a9-467a-b6a3-c18b8d86a7f8', u'port_id': u'4f81529e-a9a9-467a-b6a3-c18b8d86a7f8', u'network_type': u'vlan'}
2013-10-31 11:40:18     INFO [quantum.agent.securitygroups_rpc] Security group member updated [u'9e9f2c59-b3bf-4d1e-be0b-135f9f944e4e']
2013-10-31 11:40:18     INFO [quantum.agent.securitygroups_rpc] Refresh firewall rules
2013-10-31 11:40:18     INFO [quantum.agent.securitygroups_rpc] Provider rule updated
2013-10-31 11:40:18     INFO [quantum.agent.securitygroups_rpc] Refresh firewall rules
2013-10-31 11:52:00     INFO [quantum.agent.securitygroups_rpc] Security group member updated [u'9e9f2c59-b3bf-4d1e-be0b-135f9f944e4e']
2013-10-31 11:52:00     INFO [quantum.agent.securitygroups_rpc] Refresh firewall rules
2013-10-31 11:52:00     INFO [quantum.agent.securitygroups_rpc] Security group member updated [u'9e9f2c59-b3bf-4d1e-be0b-135f9f944e4e']
2013-10-31 11:52:00     INFO [quantum.agent.securitygroups_rpc] Refresh firewall rules
2013-10-31 12:31:35     INFO [quantum.agent.securitygroups_rpc] Security group rule updated [u'9e9f2c59-b3bf-4d1e-be0b-135f9f944e4e']
2013-10-31 12:31:35     INFO [quantum.agent.securitygroups_rpc] Refresh firewall rules
2013-10-31 12:41:07     INFO [quantum.common.config] Logging enabled!
2013-10-31 12:41:08     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Mapping physical network inter-vlan to bridge br-eth3
2013-10-31 12:41:09     INFO [quantum.openstack.common.rpc.impl_qpid] Connected to AMQP server on 10.35.160.17:5672
2013-10-31 12:41:10     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Agent initialized successfully, now running...
2013-10-31 12:41:10     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Agent out of sync with plugin!
2013-10-31 12:41:10     INFO [quantum.openstack.common.rpc.impl_qpid] Connected to AMQP server on 10.35.160.17:5672
2013-10-31 12:54:30     INFO [quantum.common.config] Logging enabled!
2013-10-31 12:54:30     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Mapping physical network inter-vlan to bridge br-eth3
2013-10-31 12:54:32     INFO [quantum.openstack.common.rpc.impl_qpid] Connected to AMQP server on 10.35.160.17:5672
2013-10-31 12:54:32     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Agent initialized successfully, now running...
2013-10-31 12:54:32     INFO [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Agent out of sync with plugin!
2013-10-31 12:54:32     INFO [quantum.openstack.common.rpc.impl_qpid] Connected to AMQP server on 10.35.160.17:5672
Comment 4 Ofer Blaut 2013-11-04 02:59:31 EST
Can't validate the bug , it is not working
Comment 9 Terry Wilson 2013-11-04 23:13:02 EST
Ofer: the steps to reproduce are how to produce the error artificially, in a way that would never happen normally. Testing shouldn't be by the above steps to reproduce, but instead should just verify that normal startup doesn't fail. The fix for this was to make sure that the scripts, when running with their normal priorities, don't occasionally fail due to the race condition that existed because daemon launched the OVS cleanup utility in a non-blocking way which could break nova if it ended up running afterward.

As it was a race condition, the only way to really try it out would be to go through lots of reboots. This shouldn't have been removed from z-stream in my opinion.
Comment 10 Ofer Blaut 2013-11-05 01:03:03 EST
Hi Terry, the normal start up works, i just followed the reproduce steps as required . 

As i mentioned in my comments from "comment from Ofer Blaut on 2013-10-23" i was not sure about the steps.

Since our nova.conf default is "resume_guests_state_on_host_boot=False"

I will change it to True and reboot the hosts to check it out .
Comment 11 Ofer Blaut 2013-11-05 08:01:42 EST
I have tried few times with start_guests_on_host_boot=True

Can't get the VMS UP after boot
Comment 12 Ofer Blaut 2013-11-19 13:38:14 EST
I have just checked it, The reproduction steps are

1. stop ovs-agent
2. stop nova-compute
3. run cleanup
4. start ovs-agent
5. start nova-compute

Only after restarting nova compute the interfaces are back
Comment 13 Ofer Blaut 2014-01-22 10:12:25 EST
already tested ( see comment 12 )
Comment 17 errata-xmlrpc 2014-01-30 14:49:50 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0110.html

Note You need to log in before you can comment on or make changes to this bug.