Red Hat Bugzilla – Bug 1316249
Neutron Port IP address pingable even when interface on overcloud node is down. Blocks interface from coming up
Last modified: 2016-04-26 14:18:09 EDT
Description of problem: Neutron Port IP address pingable even when interface on overcloud node is down. Blocks interface from coming up as we are getting error message that IP address is already in use.
Version-Release number of selected component (if applicable): 7.2
How reproducible: Unknown at this time. We believe a failure in the overcloud update process has left neutron in an undesirable state.
Steps to Reproduce:
Actual results: neutron port is responding to ping despite being down. Since neutron on the undercloud is responding to ping, we cannot bring up the interface on the overcloud controller.
Expected results: When interface on overcloud controller is down, the ip address associated to the neutron port should not respond to ping.
Additional info: Upstream switch is Big Switch.
Please add reproduction steps and more details. What Neutron port? What interface on overcloud node? How come bringing up the interface is not possible, what error are you getting? What Neutron plugin / ml2 mech driver are you using?
Note: After about 30 minutes of failing - the neutron ip finally stopped responding to ping
Created attachment 1134626 [details]
1) On undercloud node - run neutron port list
2) find port that corresponds to ip address on your overcloud controller
3) log onto overcloud controller
4) ifdown that interface on the undercloud controller
5) try to ping that ip. It should not respond. If it does you have recreated the customer's situation and if you try to ifup the interface, network.service will tell you no, as that IP address is in use elsewhere.
Created attachment 1134627 [details]
Created attachment 1134628 [details]
undercloud plugin ini
(In reply to Assaf Muller from comment #2)
> Please add reproduction steps and more details. What Neutron port? What
> interface on overcloud node? How come bringing up the interface is not
> possible, what error are you getting? What Neutron plugin / ml2 mech driver
> are you using?
The port we are referring to is a port created on the undercloud controller which is the ip of the storage interface on an overcloud controller node.
When attempting to bring up the interface on that overcloud controller node which is assigned that ip address, we get an error message that the ip is in use by another host.
Mechanism drivers we are enabling in plugin.ini
Eventually the interface stopped pinging after rebooting the undercloud/director node and we were able to successfully bring up the interface on the overcloud node which the ip belonged to.
I understand rebooting the node solved the issue.
Note this is not the provisioning interface, its occurring on storage interface
Moving to Director. If ifdown doesn't remove an IP from an interface, that's really unrelated to Neutron. It sounds like a networking issue, just not a Neutron one.
The problem is that cannot ifup the interface. We cannot find any interface where the ip is assigned, yet os still reports ip is in use on another host when we try to ifup.
Northbound big switch
member port-group BAGG100 vlan 502
member port-group any vlan 502
member switch any interface any vlan 502
ARP TABLE FROM BCF
TPA-BCF-CRL-02# show tenant OPENSTACK-MANAGEMENT segment VL502-Storage-Network-172.17.1.0-24 endpoint
# Tenant Segment Name MAC IP Address IP State Attachment Point Attach Point State VLAN State
1 OPENSTACK-MANAGEMENT VL502-Storage-Network-172.17.1.0-24 14:18:77:30:1b:5a (Dell) 172.17.1.202 learned-L2 Only crl-u32-UTL32 learned 502 Active
2 OPENSTACK-MANAGEMENT VL502-Storage-Network-172.17.1.0-24 a6:18:86:69:f2:31 172.17.1.203 learned-L2 Only crl-u7-com7 learned 502 Active
3 OPENSTACK-MANAGEMENT VL502-Storage-Network-172.17.1.0-24 f6:86:a5:c5:f2:d2 172.17.1.204 learned-L2 Only crl-u9-com9 learned 502 Active