Bug 1474025 - No connectivity to An instance Floating IP after Restarting the compute node
No connectivity to An instance Floating IP after Restarting the compute node
Product: Red Hat OpenStack
Classification: Red Hat
Component: opendaylight (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
high Severity high
: ---
: 12.0 (Pike)
Assigned To: Sridhar Gaddam
Itzik Brown
: Triaged
Depends On:
  Show dependency treegraph
Reported: 2017-07-23 04:29 EDT by Itzik Brown
Modified: 2017-12-14 04:45 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-12-14 04:45:33 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
OpenDaylight Bug 8877 None None None 2017-07-23 04:29 EDT

  None (edit)
Description Itzik Brown 2017-07-23 04:29:49 EDT
Description of problem:
Deploying OpenStack Pike with Carbon SR1.
After restarting a compute node and launching an instance the Instance doesn't get an IP from DHCP.
After restarting Open vSwitch and running the DHCP client again the Instance gets an IP.

It seems that the following flow is missing 
table=0, n_packets=0, n_bytes=0, priority=4,in_port=2,vlan_tci=0x0000/0x1fff actions=write_metadata:0x180000000000/0xffffff0000000001,goto_table:17

Version-Release number of selected component (if applicable):
OpenStack Pike
Open vSwitch 2.7

How reproducible:

Steps to Reproduce:
1. Restart the compute node 
2. Launch an instance on the above compute node
3. Verify the instance is not getting an IP
4. Restart the openvswitch on the compute node
5. Restart the DHCP client and verify the instance gets an IP.

Actual results:

Expected results:

Additional info:
Comment 1 Sridhar Gaddam 2017-08-01 11:37:06 EDT
On debugging this issue closely, it appears like some race condition in ODL Controller.

Steps to Reproduce:
1. Restart the compute node 
2. Launch an instance on the compute node
3. You can observe that the instance initially stays in "spawning" state and then transitions to "error" state.
4. Restart the openvswitch on the compute node
5. Launch a new instance and it would boot successfully.

Basically, when we issue the reboot on the compute node, ODL identifies that the node is idle and triggers the disconnection chain. 
But, while this is going on, when the Compute node comes up, we could see that there is a race condition between the cleanup events and the events related to the node reconciliation.

In this process, we could see that finally the Compute node is deleted from the operational store [#] eventhough its connected to the controller. 
Since the node info is deleted from the datastore, the sideeffect is that port-binding fails and we will be unable to spawn new VMs until we restart the OVS Switch on the Compute node.
Following[@] is a SNAP of the karaf logs which show this sequence.

Some additional notes:
Incase, the compute node comes up with some delay (i.e., after the cleanup is properly done in ODL) this issue (i.e., step3 above) is not seen.

[#] 2017-08-01 07:48:16,660 | INFO  | lt-dispatcher-49 | OvsdbConnectionManager           | 289 - org.opendaylight.ovsdb.southbound-impl - 1.4.1.Carbon-redhat-1 | Entity{type='ovsdb', id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}]/node/node[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/e9806896-8dc2-4f17-83ea-c1c957608915}]} has no owner, cleaning up the operational data store
[@] https://gist.github.com/sridhargaddam/3761ef080e11f2dd2429c8d7016ae6d0
Comment 4 Itzik Brown 2017-09-26 06:03:33 EDT
Checked with opendaylight-6.2.0-0.1.20170921snap729.el7.noarch.

After the compute node restart, launching again the instance - I get an IP.
The problem now is that I don't have connectivity to the instance's FIP.
Comment 6 Itzik Brown 2017-12-14 04:45:33 EST
Opening a new bug for the FIP issue.

Note You need to log in before you can comment on or make changes to this bug.