Bug 1303758 - Compute nodes get IPv6 addresses and default route installed on the qbr interfaces via RAs from the tenant network
Compute nodes get IPv6 addresses and default route installed on the qbr inter...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity high
: y3
: 7.0 (Kilo)
Assigned To: Jiri Stransky
Marius Cornea
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-01 16:10 EST by Marius Cornea
Modified: 2016-02-18 11:52 EST (History)
10 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-0.8.6-117.el7ost
Doc Type: Bug Fix
Doc Text:
Compute nodes detected IPv6 router announcements (RA) on the Overcloud's IPv6-based Tenant network. This caused the Compute nodes to receive IPv6 addresses and default routes on the 'qbr' interface. This fix sets the 'net.ipv6.conf.default.disable_ipv6' kernel parameter to 1 on all nodes. This disables automatic configuration from RAs and allows the director to define the IPv6 addresses and default route.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-18 11:52:11 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Bugzilla 1305124 None None None 2016-02-06 06:49 EST
Red Hat Bugzilla 1305128 None None None 2016-02-06 06:48 EST
OpenStack gerrit 277182 None None None 2016-02-09 12:02 EST

  None (edit)
Description Marius Cornea 2016-02-01 16:10:01 EST
Description of problem:
When using vlan tenat networks the compute nodes get IPv6 addresses and default route installed on the qbr interfaces via RAs from the tenant network.  

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-0.8.6-112.el7ost.noarch

How reproducible:
100% 

Steps to Reproduce:
1. Deploy IPv6 enabled overcloud in virtual environment

2. Create vlan provider network:

neutron net-create provider-1000 --shared  --provider:physical_network datacentre --provider:network_type vlan --provider:segmentation_id 1000
neutron subnet-create provider-1000 10.0.0.0/24 --name provider-1000-subnet-ipv4 --gateway 10.0.0.1
neutron subnet-create provider-1000 2001:db1::/64 --name provider-1000-subnet-ipv6 --gateway 2001:db1::1 --ipv6-ra-mode slaac --ipv6-address-mode slaac --ip-version 6

3. On the undercloud create a vlan interface that matches the vlan id of the network created in step 2:

ovs-vsctl add-port br-ctlplane vlan1000 tag=1000 -- set interface vlan1000 type=internal
ip link set dev vlan1000 up
ip addr add 2001:db1::1/64 dev vlan1000

4. Configure radvd for the vlan interface configured in step 3

cat /etc/radvd.conf
interface vlan1000 { 
        AdvSendAdvert on;
        MinRtrAdvInterval 3; 
        MaxRtrAdvInterval 10;
        prefix 2001:db1::/64  { 
                AdvOnLink on; 
                AdvAutonomous on; 
                AdvRouterAddr on; 
        };
};

5. Boot instance with nic attached to provider-1000 network

Actual results:

The compute node gets an IPv6 address configured on the qbr bridge associated to the instance:

[root@overcloud-compute-0 ~]# ip a s dev qbr232149d8-52 | grep inet6
    inet6 2001:db1::3cce:d7ff:fe2d:40b5/64 scope global mngtmpaddr dynamic 
    inet6 fe80::3cce:d7ff:fe2d:40b5/64 scope link 

and a default route as well:

[root@overcloud-compute-0 ~]# ip -6 r | grep default
default via fe80::486b:dcff:fe69:215f dev qbr232149d8-52  proto ra  metric 1024  expires 25sec hoplimit 64

Expected results:
The compute nodes do not accept the RAs coming on the tenant networks.
Comment 1 Marius Cornea 2016-02-02 20:09:17 EST
FWIW this also shows up when the RAs are originated by neutron routers:

neutron router-create router01
neutron net-create provider-2000 --provider:physical_network datacentre --provider:network_type vlan --provider:segmentation_id 2000
neutron subnet-create provider-2000 2001:db2::/64 --name provider-2000-subnet-ipv6 --gateway 2001:db2::1 --ipv6-ra-mode slaac --ipv6-address-mode slaac --ip-version 6
neutron router-interface-add router01 provider-2000-subnet-ipv6

neutron router-create router02
neutron net-create provider-3000 --provider:physical_network datacentre --provider:network_type vlan --provider:segmentation_id 3000
neutron subnet-create provider-3000 2001:db3::/64 --name provider-3000-subnet-ipv6 --gateway 2001:db3::1 --ipv6-ra-mode slaac --ipv6-address-mode slaac --ip-version 6
neutron router-interface-add router02 provider-3000-subnet-ipv6

After launching 2 VMs in each of the networks created in the previous step we get the following on the compute node:

[root@overcloud-compute-0 ~]# ip a s dev qbr938b517a-41
9: qbr938b517a-41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 8a:ab:6e:5b:17:d4 brd ff:ff:ff:ff:ff:ff
    inet6 2001:db2::88ab:6eff:fe5b:17d4/64 scope global mngtmpaddr dynamic 
       valid_lft 86399sec preferred_lft 14399sec
    inet6 fe80::88ab:6eff:fe5b:17d4/64 scope link 
       valid_lft forever preferred_lft forever
[root@overcloud-compute-0 ~]# ip a s dev qbr4f3f1014-20
13: qbr4f3f1014-20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 0a:aa:63:08:f3:a2 brd ff:ff:ff:ff:ff:ff
    inet6 2001:db3::8aa:63ff:fe08:f3a2/64 scope global mngtmpaddr dynamic 
       valid_lft 86397sec preferred_lft 14397sec
    inet6 fe80::8aa:63ff:fe08:f3a2/64 scope link 
       valid_lft forever preferred_lft forever

[root@overcloud-compute-0 ~]# ip -6 r | grep default
default via fe80::f816:3eff:fed4:5222 dev qbr938b517a-41  proto ra  metric 1024  expires 29sec hoplimit 64
default via fe80::f816:3eff:fee9:89a8 dev qbr4f3f1014-20  proto ra  metric 1024  expires 26sec hoplimit 64

[root@overcloud-controller-0 ~]# ip netns exec qrouter-187d55e9-55e5-44c8-8358-ae9bdaa119f0 ip a | grep inet6
    inet6 ::1/128 scope host 
    inet6 2001:db3::1/64 scope global 
    inet6 fe80::f816:3eff:fee9:89a8/64 scope link 
[root@overcloud-controller-0 ~]# ip netns exec qrouter-71bc2d72-1b55-484c-8543-41071119110e ip a | grep inet6
    inet6 ::1/128 scope host 
    inet6 2001:db2::1/64 scope global 
    inet6 fe80::f816:3eff:fed4:5222/64 scope link
Comment 2 Dan Sneddon 2016-02-03 06:17:16 EST
A few things about this:

If the Compute nodes had an IPv6 interface with external access, then that network would have a much better metric than 1024. The metric of 1024 shows that these are really considered routes of last resort.

Since the compute nodes don't have a default route for IPv6, this would add one, but since we don't configure any network services to use IPv6, this should be a harmless artifact. We might want to implement iptables on the Compute nodes to prevent unwanted IPv6 communication. There may also be options for disabling automatic router configuration on interfaces with link-local addresses.

If you don't want this behavior, then you can not use radvd on the Undercloud or a Neutron router. From the OpenStack IPv6 guide [1]:

"In the absence of an upstream RA support, ipv6_gateway flag can be set with the external router gateway LLA in the neutron L3 agent configuration file."

Here is some more info on ipv6_gateway [2].

However, the problem with this approach is that the instances would get link-local addresses. The router gateway address must be the link-local address on the router, not a globally addressable IPv6 address on the router.

Still, you should be able to ping a link-local VM from a link-local address on the under cloud, even if you haven't assigned an IP to the Undercloud VLAN or turned on radvd.

[1] - http://docs.openstack.org/liberty/networking-guide/adv_config_ipv6.html
[2] - http://docs.openstack.org/kilo/config-reference/content/networking-options-l3_agent.html
Comment 3 James Slagle 2016-02-03 09:43:08 EST
closing not a bug per dan
Comment 4 Marius Cornea 2016-02-03 14:16:59 EST
The qbrs are bridges for iptables filtering so I don't think they require an IP address. 

In my deployment (networks created according to comment#1) the VMs in the tenant network get direct connectivity to the hypervisor which can pose to be a security issue. If you know the global address assigned to the qbr interface then from inside the VM you can access the VNC consoles of *all* the instances running on the same host without authentication.

#this is the host
[root@overcloud-compute-0 ~]# ip a s dev qbr95384bae-5b | grep inet6
    inet6 2001:db2::c41f:99ff:fe38:b85d/64 scope global mngtmpaddr dynamic 
    inet6 fe80::c41f:99ff:fe38:b85d/64 scope link 

#this is the VM running on the host
[root@instance ~]# ip a s dev eth0 | grep inet6
    inet6 2001:db2::f816:3eff:fefa:1aff/64 scope global mngtmpaddr dynamic 
    inet6 fe80::f816:3eff:fefa:1aff/64 scope link 
[root@instance ~]# curl http://[2001:db2::c41f:99ff:fe38:b85d]:5900
RFB 003.008
[root@instance ~]# curl http://[2001:db2::c41f:99ff:fe38:b85d]:5901
RFB 003.008
Comment 5 Dan Sneddon 2016-02-04 10:57:45 EST
(In reply to Marius Cornea from comment #4)
> The qbrs are bridges for iptables filtering so I don't think they require an
> IP address. 
> 
> In my deployment (networks created according to comment#1) the VMs in the
> tenant network get direct connectivity to the hypervisor which can pose to
> be a security issue. If you know the global address assigned to the qbr
> interface then from inside the VM you can access the VNC consoles of *all*
> the instances running on the same host without authentication.
> 
> #this is the host
> [root@overcloud-compute-0 ~]# ip a s dev qbr95384bae-5b | grep inet6
>     inet6 2001:db2::c41f:99ff:fe38:b85d/64 scope global mngtmpaddr dynamic 
>     inet6 fe80::c41f:99ff:fe38:b85d/64 scope link 
> 
> #this is the VM running on the host
> [root@instance ~]# ip a s dev eth0 | grep inet6
>     inet6 2001:db2::f816:3eff:fefa:1aff/64 scope global mngtmpaddr dynamic 
>     inet6 fe80::f816:3eff:fefa:1aff/64 scope link 
> [root@instance ~]# curl http://[2001:db2::c41f:99ff:fe38:b85d]:5900
> RFB 003.008
> [root@instance ~]# curl http://[2001:db2::c41f:99ff:fe38:b85d]:5901
> RFB 003.008

Even if the interfaces don't have an IP address, in fact even if we turn off IP auto configuration, the system will still accept RAs by default and create routes.

In order to turn this behavior on/off, you can edit /proc/sys/net/ipv6/conf/default/accept_ra to be "0" instead of "1".

You can also turn it on/off on a per-interface basis, e.g.:
echo "1" > /proc/sys/net/ipv6/conf/eth0/accept_ra
Comment 8 Dan Sneddon 2016-02-06 06:49:54 EST
Linked the BZs for implementing a fix on 7.3 and 8.0.
Comment 9 Dan Sneddon 2016-02-07 10:47:18 EST
This should be fixed both upstream and downstream. Here is the downstream fix proposed:

https://code.engineering.redhat.com/gerrit/#/c/67206/
Comment 11 Marius Cornea 2016-02-12 08:30:44 EST
[root@overcloud-compute-0 ~]# ip a s dev qbr203e08a5-42
13: qbr203e08a5-42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 3a:27:0e:61:8c:e1 brd ff:ff:ff:ff:ff:ff
Comment 13 errata-xmlrpc 2016-02-18 11:52:11 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0264.html

Note You need to log in before you can comment on or make changes to this bug.