Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1316249

Summary: Neutron Port IP address pingable even when interface on overcloud node is down. Blocks interface from coming up
Product: Red Hat OpenStack Reporter: Chris Paquin <cpaquin>
Component: openstack-neutronAssignee: Assaf Muller <amuller>
Status: CLOSED NOTABUG QA Contact: Toni Freger <tfreger>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: amuller, chrisw, cpaquin, dbecker, jay.cromer, liko, mburns, morazi, nyechiel, rhel-osp-director-maint, yeylon
Target Milestone: ---Keywords: Reopened
Target Release: 8.0 (Liberty)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-14 18:51:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
plugin.ini
none
overcloud neutron.conf
none
undercloud plugin ini none

Description Chris Paquin 2016-03-09 18:59:37 UTC
Description of problem: Neutron Port IP address pingable even when interface on overcloud node is down. Blocks interface from coming up as we are getting error message that IP address is already in use. 


Version-Release number of selected component (if applicable): 7.2


How reproducible: Unknown at this time. We believe a failure in the overcloud update process has left neutron in an undesirable state.  


Steps to Reproduce:
1.
2.
3.

Actual results: neutron port is responding to ping despite being down. Since neutron on the undercloud is responding to ping, we cannot bring up the interface on the overcloud controller.


Expected results: When interface on overcloud controller is down, the ip address associated to the neutron port should not respond to ping. 


Additional info: Upstream switch is Big Switch.

Comment 2 Assaf Muller 2016-03-09 19:04:24 UTC
Please add reproduction steps and more details. What Neutron port? What interface on overcloud node? How come bringing up the interface is not possible, what error are you getting? What Neutron plugin / ml2 mech driver are you using?

Comment 3 Chris Paquin 2016-03-09 19:16:08 UTC
Note: After about 30 minutes of failing - the neutron ip finally stopped responding to ping

Comment 5 Chris Paquin 2016-03-09 19:28:29 UTC
Created attachment 1134626 [details]
plugin.ini

Comment 6 Chris Paquin 2016-03-09 19:33:00 UTC
1) On undercloud node - run neutron  port list

2) find port that corresponds to ip address on your overcloud controller

3) log onto overcloud controller

4) ifdown that interface on the undercloud controller

5) try to ping that ip. It should not respond. If it does you have recreated the customer's situation and if you try to ifup the interface, network.service will tell you no, as that IP address is in use elsewhere.

Comment 7 Chris Paquin 2016-03-09 19:37:07 UTC
Created attachment 1134627 [details]
overcloud neutron.conf

Comment 8 Chris Paquin 2016-03-09 19:38:10 UTC
Created attachment 1134628 [details]
undercloud plugin ini

Comment 9 Jay Cromer 2016-03-09 19:45:19 UTC
(In reply to Assaf Muller from comment #2)
> Please add reproduction steps and more details. What Neutron port? What
> interface on overcloud node? How come bringing up the interface is not
> possible, what error are you getting? What Neutron plugin / ml2 mech driver
> are you using?

The port we are referring to is a port created on the undercloud controller which is the ip of the storage interface on an overcloud controller node.

When attempting to bring up the interface on that overcloud controller node which is assigned that ip address, we get an error message that the ip is in use by another host.

Mechanism drivers we are enabling in plugin.ini
mechanism_drivers =openvswitch,linuxbridge

Comment 10 Jay Cromer 2016-03-09 19:59:53 UTC
Eventually the interface stopped pinging after rebooting the undercloud/director node and we were able to successfully bring up the interface on the overcloud node which the ip belonged to.

Comment 11 Assaf Muller 2016-03-09 20:12:51 UTC
I understand rebooting the node solved the issue.

Comment 14 Chris Paquin 2016-03-14 17:53:03 UTC
Note this is not the provisioning interface, its occurring on storage interface

Comment 15 Assaf Muller 2016-03-14 18:21:01 UTC
Moving to Director. If ifdown doesn't remove an IP from an interface, that's really unrelated to Neutron. It sounds like a networking issue, just not a Neutron one.

Comment 16 Jay Cromer 2016-03-14 18:24:04 UTC
The problem is that cannot ifup the interface.  We cannot find any interface where the ip is assigned, yet os still reports ip is in use on another host when we try to ifup.

Comment 17 Chris Paquin 2016-03-14 18:51:50 UTC
Northbound big switch

tenant OPENSTACK-MANAGEMENT
 segment VL502-Storage-Network-172.17.1.0-24
   member port-group BAGG100 vlan 502
   member port-group any vlan 502
   member switch any interface any vlan 502


ARP TABLE FROM BCF

TPA-BCF-CRL-02# show tenant OPENSTACK-MANAGEMENT segment VL502-Storage-Network-172.17.1.0-24 endpoint
# Tenant               Segment                             Name MAC                      IP Address   IP State        Attachment Point Attach Point State VLAN State
-|--------------------|-----------------------------------|----|------------------------|------------|---------------|----------------|------------------|----|------|
1 OPENSTACK-MANAGEMENT VL502-Storage-Network-172.17.1.0-24      14:18:77:30:1b:5a (Dell) 172.17.1.202 learned-L2 Only crl-u32-UTL32    learned            502  Active
2 OPENSTACK-MANAGEMENT VL502-Storage-Network-172.17.1.0-24      a6:18:86:69:f2:31        172.17.1.203 learned-L2 Only crl-u7-com7      learned            502  Active
3 OPENSTACK-MANAGEMENT VL502-Storage-Network-172.17.1.0-24      f6:86:a5:c5:f2:d2        172.17.1.204 learned-L2 Only crl-u9-com9      learned            502  Active