Bug 1316249 - Neutron Port IP address pingable even when interface on overcloud node is down. Blocks interface from coming up
Neutron Port IP address pingable even when interface on overcloud node is dow...
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
7.0 (Kilo)
x86_64 Linux
unspecified Severity high
: ---
: 8.0 (Liberty)
Assigned To: Assaf Muller
Toni Freger
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-09 13:59 EST by Chris Paquin
Modified: 2016-04-26 14:18 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-14 14:51:50 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
plugin.ini (4.01 KB, text/plain)
2016-03-09 14:28 EST, Chris Paquin
no flags Details
overcloud neutron.conf (36.04 KB, text/plain)
2016-03-09 14:37 EST, Chris Paquin
no flags Details
undercloud plugin ini (3.99 KB, text/plain)
2016-03-09 14:38 EST, Chris Paquin
no flags Details

  None (edit)
Description Chris Paquin 2016-03-09 13:59:37 EST
Description of problem: Neutron Port IP address pingable even when interface on overcloud node is down. Blocks interface from coming up as we are getting error message that IP address is already in use. 


Version-Release number of selected component (if applicable): 7.2


How reproducible: Unknown at this time. We believe a failure in the overcloud update process has left neutron in an undesirable state.  


Steps to Reproduce:
1.
2.
3.

Actual results: neutron port is responding to ping despite being down. Since neutron on the undercloud is responding to ping, we cannot bring up the interface on the overcloud controller.


Expected results: When interface on overcloud controller is down, the ip address associated to the neutron port should not respond to ping. 


Additional info: Upstream switch is Big Switch.
Comment 2 Assaf Muller 2016-03-09 14:04:24 EST
Please add reproduction steps and more details. What Neutron port? What interface on overcloud node? How come bringing up the interface is not possible, what error are you getting? What Neutron plugin / ml2 mech driver are you using?
Comment 3 Chris Paquin 2016-03-09 14:16:08 EST
Note: After about 30 minutes of failing - the neutron ip finally stopped responding to ping
Comment 5 Chris Paquin 2016-03-09 14:28 EST
Created attachment 1134626 [details]
plugin.ini
Comment 6 Chris Paquin 2016-03-09 14:33:00 EST
1) On undercloud node - run neutron  port list

2) find port that corresponds to ip address on your overcloud controller

3) log onto overcloud controller

4) ifdown that interface on the undercloud controller

5) try to ping that ip. It should not respond. If it does you have recreated the customer's situation and if you try to ifup the interface, network.service will tell you no, as that IP address is in use elsewhere.
Comment 7 Chris Paquin 2016-03-09 14:37 EST
Created attachment 1134627 [details]
overcloud neutron.conf
Comment 8 Chris Paquin 2016-03-09 14:38 EST
Created attachment 1134628 [details]
undercloud plugin ini
Comment 9 Jay Cromer 2016-03-09 14:45:19 EST
(In reply to Assaf Muller from comment #2)
> Please add reproduction steps and more details. What Neutron port? What
> interface on overcloud node? How come bringing up the interface is not
> possible, what error are you getting? What Neutron plugin / ml2 mech driver
> are you using?

The port we are referring to is a port created on the undercloud controller which is the ip of the storage interface on an overcloud controller node.

When attempting to bring up the interface on that overcloud controller node which is assigned that ip address, we get an error message that the ip is in use by another host.

Mechanism drivers we are enabling in plugin.ini
mechanism_drivers =openvswitch,linuxbridge
Comment 10 Jay Cromer 2016-03-09 14:59:53 EST
Eventually the interface stopped pinging after rebooting the undercloud/director node and we were able to successfully bring up the interface on the overcloud node which the ip belonged to.
Comment 11 Assaf Muller 2016-03-09 15:12:51 EST
I understand rebooting the node solved the issue.
Comment 14 Chris Paquin 2016-03-14 13:53:03 EDT
Note this is not the provisioning interface, its occurring on storage interface
Comment 15 Assaf Muller 2016-03-14 14:21:01 EDT
Moving to Director. If ifdown doesn't remove an IP from an interface, that's really unrelated to Neutron. It sounds like a networking issue, just not a Neutron one.
Comment 16 Jay Cromer 2016-03-14 14:24:04 EDT
The problem is that cannot ifup the interface.  We cannot find any interface where the ip is assigned, yet os still reports ip is in use on another host when we try to ifup.
Comment 17 Chris Paquin 2016-03-14 14:51:50 EDT
Northbound big switch

tenant OPENSTACK-MANAGEMENT
 segment VL502-Storage-Network-172.17.1.0-24
   member port-group BAGG100 vlan 502
   member port-group any vlan 502
   member switch any interface any vlan 502


ARP TABLE FROM BCF

TPA-BCF-CRL-02# show tenant OPENSTACK-MANAGEMENT segment VL502-Storage-Network-172.17.1.0-24 endpoint
# Tenant               Segment                             Name MAC                      IP Address   IP State        Attachment Point Attach Point State VLAN State
-|--------------------|-----------------------------------|----|------------------------|------------|---------------|----------------|------------------|----|------|
1 OPENSTACK-MANAGEMENT VL502-Storage-Network-172.17.1.0-24      14:18:77:30:1b:5a (Dell) 172.17.1.202 learned-L2 Only crl-u32-UTL32    learned            502  Active
2 OPENSTACK-MANAGEMENT VL502-Storage-Network-172.17.1.0-24      a6:18:86:69:f2:31        172.17.1.203 learned-L2 Only crl-u7-com7      learned            502  Active
3 OPENSTACK-MANAGEMENT VL502-Storage-Network-172.17.1.0-24      f6:86:a5:c5:f2:d2        172.17.1.204 learned-L2 Only crl-u9-com9      learned            502  Active

Note You need to log in before you can comment on or make changes to this bug.