Bug 1270910 - IP address from external subnet gets assigned to br-ex when using default single-nic-vlans templates
IP address from external subnet gets assigned to br-ex when using default sin...
Status: CLOSED ERRATA
Product: RDO
Classification: Community
Component: rdo-manager (Show other bugs)
Liberty
Unspecified Unspecified
high Severity high
: GA
: Liberty
Assigned To: Dan Sneddon
Shai Revivo
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-12 11:53 EDT by Marius Cornea
Modified: 2017-07-04 11:02 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-04 11:02:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
network environment (1.30 KB, text/plain)
2015-10-12 11:53 EDT, Marius Cornea
no flags Details

  None (edit)
Description Marius Cornea 2015-10-12 11:53:59 EDT
Created attachment 1082050 [details]
network environment

Description of problem:
IP address from external subnet gets assigned to br-ex when using default single-nic-vlans templates

Version-Release number of selected component (if applicable):
python-heatclient-0.8.1-dev1.el7.centos.noarch
openstack-heat-engine-5.0.0.0-rc2.dev1.el7.centos.noarch
openstack-heat-common-5.0.0.0-rc2.dev1.el7.centos.noarch
openstack-heat-api-cfn-5.0.0.0-rc2.dev1.el7.centos.noarch
openstack-heat-api-5.0.0.0-rc2.dev1.el7.centos.noarch
openstack-heat-templates-0.0.1-dev381.el7.centos.noarch
openstack-heat-api-cloudwatch-5.0.0.0-rc2.dev1.el7.centos.noarch
openstack-tripleo-heat-templates-0.8.7-dev277.el7.centos.noarch
os-net-config-0.1.5-dev12.el7.centos.noarch

How reproducible:
100%

Steps to Reproduce:
Following the docs at: https://repos.fedorapeople.org/repos/openstack-m/rdo-manager-docs/liberty/

1. Deploy overcloud with one controller, one compute with network isolation
openstack overcloud deploy --templates ~/templates/my-overcloud -e ~/templates/my-overcloud/environments/network-isolation.yaml -e ~/templates/network-environment.yaml --control-scale 1 --compute-scale 1 --libvirt-type qemu

Actual results:
IP address from the external network (public_virtual_ip neutron port) gets assigned to the br-ex device on the overcloud controller. 
[stack@instack ~]$ ssh heat-admin@192.0.2.13 'sudo ip a s dev br-ex | grep inet'
    inet 192.0.2.13/24 brd 192.0.2.255 scope global br-ex
    inet 192.0.2.12/32 scope global br-ex
    inet 172.16.23.10/32 scope global br-ex
    inet6 fe80::231:d1ff:fe11:6eb1/64 scope link 
[stack@instack ~]$ ssh heat-admin@192.0.2.13 'sudo ip a s dev vlan10 | grep inet'
    inet 172.16.23.11/24 brd 172.16.23.255 scope global vlan10
    inet6 fe80::d8a7:5bff:fe80:a097/64 scope link 

Expected results:
I'm expecting the public_virtual_ip to be assigned on the vlan10 device. 

Additional info:
Attaching network-environment.yaml file.
Comment 1 Marius Cornea 2015-10-12 12:56:46 EDT
Note that when passing /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml as environment file I got the IP set as expected:

openstack overcloud deploy --templates ~/templates/my-overcloud -e ~/templates/my-overcloud/environments/network-isolation.yaml -e ~/templates/network-environment.yaml --control-scale 1 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml

[stack@instack ~]$ ssh heat-admin@192.0.2.16 'sudo ip a s dev br-ex | grep inet'
    inet 192.0.2.16/24 brd 192.0.2.255 scope global br-ex
    inet 192.0.2.15/32 brd 192.0.2.255 scope global br-ex
    inet6 fe80::231:d1ff:fe11:6eb1/64 scope link 
[stack@instack ~]$ ssh heat-admin@192.0.2.16 'sudo ip a s dev vlan10 | grep inet'
    inet 172.16.23.11/24 brd 172.16.23.255 scope global vlan10
    inet 172.16.23.10/32 brd 172.16.23.255 scope global vlan10
    inet6 fe80::4c0f:deff:fe2b:d574/64 scope link
Comment 2 Dan Sneddon 2015-10-13 13:13:28 EDT
I'm not sure what is going on here. I just did a deployment over the weekend with the single-nic-with-vlans templates, and this didn't happen. I'm running it again just to be sure.
Comment 3 Dan Sneddon 2015-10-13 13:28:05 EDT
(In reply to Dan Sneddon from comment #2)
> I'm not sure what is going on here. I just did a deployment over the weekend
> with the single-nic-with-vlans templates, and this didn't happen. I'm
> running it again just to be sure.

Nevermind, I was not using the latest Liberty code. This is probably a new problem which has crept in upstream.
Comment 4 John Trowbridge 2015-10-14 09:59:48 EDT
@Dan, is network isolation even supposed to work without pacemaker?

see comment 1: https://bugzilla.redhat.com/show_bug.cgi?id=1270910#c1
Comment 5 Dan Sneddon 2015-10-14 19:55:54 EDT
(In reply to John Trowbridge from comment #4)
> @Dan, is network isolation even supposed to work without pacemaker?
> 
> see comment 1: https://bugzilla.redhat.com/show_bug.cgi?id=1270910#c1

Sure, if you don't have HA then Pacemaker is not required. Network Isolation doesn't have any dependencies on Pacemaker. When you use "-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml", here's what happens:

puppet-pacemaker.yaml:
  OS::TripleO::ControllerConfig: ../puppet/controller-config-pacemaker.yaml

controller-config-pacemaker.yaml:
(doesn't do much, except it includes overcloud_controller_pacemaker.pp:
        get_file: manifests/overcloud_controller_pacemaker.pp

So, the difference is probably in how overcloud_controller_pacemaker.pp instantiates the public VIP:

#######
      $public_vip = hiera('tripleo::loadbalancer::public_virtual_ip')
      if $public_vip and $public_vip != $control_vip {
        pacemaker::resource::ip { 'public_vip':
          ip_address => $public_vip,
        }
        pacemaker::constraint::base { 'public_vip-then-haproxy':
          constraint_type   => 'order',
          first_resource    => "ip-${public_vip}",
          second_resource   => 'haproxy-clone',
          first_action      => 'start',
          second_action     => 'start',
          constraint_params => 'kind=Optional',
          require => [Pacemaker::Resource::Service['haproxy'],
                      Pacemaker::Resource::Ip['public_vip']],
        }
        pacemaker::constraint::colocation { 'public_vip-with-haproxy':
          source  => "ip-${public_vip}",
          target  => 'haproxy-clone',
          score   => 'INFINITY',
          require => [Pacemaker::Resource::Service['haproxy'],
                      Pacemaker::Resource::Ip['public_vip']],
        }
      }
#######

In a non-HA environment, haproxy will bring up the VIP, due to the bind statements in /etc/haproxy.conf, for example:

[heat-admin@overcloud-controller-0 etc]$ sudo grep -r 10.0.0.4 /etc/*
/etc/dnsmasq.conf:# and this maps 192.168.0.10->192.168.0.40 to 10.0.0.10->10.0.0.40
grep: /etc/extlinux.conf: No such file or directory
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:8777 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:8776 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:9292 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:8004 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:8000 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:8003 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:80 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:35357 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:5000 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:9696 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:8773 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:6080 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:8774 
/etc/haproxy/haproxy.cfg:  bind 10.0.0.4:8080 
/etc/puppet/hieradata/vip_data.yaml:tripleo::loadbalancer::public_virtual_ip: 10.0.0.4

I haven't had a chance to test this on Liberty yet, so I'm not sure what is going on here. In older code, haproxy had those bind statements, and the VIP would come up automatically on the interface that matched the subnet. I can't imagine what we are doing in Liberty to cause that VIP to come up on the wrong interface.
Comment 7 Giulio Fidente 2015-10-15 05:46:04 EDT
Dan, Marius, in the non-HA scenario it is Keepalived bringing up the VIPs.

It requires us to pass an interface and we seem to be passing the correct variable, not sure abut its contents though, see [1].

Can you run the following two from a controller node:

sudo hiera tripleo::loadbalancer::control_virtual_interface
sudo hiera tripleo::loadbalancer::public_virtual_interface

1. https://github.com/openstack/puppet-tripleo/blob/master/manifests/loadbalancer.pp#L343
Comment 8 Marius Cornea 2015-10-15 07:28:09 EDT
Hi Giulio, it seems that br-ex is used for both:
[heat-admin@overcloud-controller-0 ~]$ sudo hiera tripleo::loadbalancer::control_virtual_interface
br-ex
[heat-admin@overcloud-controller-0 ~]$ sudo hiera tripleo::loadbalancer::public_virtual_interface
br-ex
Comment 9 Giulio Fidente 2015-10-15 11:17:13 EDT
It looks like that is just a configurable value and it is coming from the PublicVirtualInterface parameter; defaults to br-ex.

Can you try passing the appropriate interface name with something like:

parameters:
  PublicVirtualInterface: vlan10
Comment 10 Marius Cornea 2015-10-15 11:35:45 EDT
Works:

Added 'PublicVirtualInterface: vlan10' to parameter_defaults in network-environment.yaml and ran the deploy command:

[root@overcloud-controller-0 ~]# sudo hiera tripleo::loadbalancer::control_virtual_interface
br-ex
[root@overcloud-controller-0 ~]# sudo hiera tripleo::loadbalancer::public_virtual_interface
vlan10
Comment 13 Bob Fournier 2017-02-28 14:46:35 EST
This has been fixed - https://review.openstack.org/407196.

Note You need to log in before you can comment on or make changes to this bug.