Bug 1643135
| Summary: | OSP 13 - OVS/ml2 - ipv6 forwarding is broken in HA neutron routers due to incorrect ipv6 kernel forwarding parameters | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Matt Flusche <mflusche> | |
| Component: | openstack-neutron | Assignee: | Brian Haley <bhaley> | |
| Status: | CLOSED ERRATA | QA Contact: | Roee Agiman <ragiman> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 13.0 (Queens) | CC: | amuller, bhaley, chrisw, jschluet | |
| Target Milestone: | --- | Keywords: | Triaged, ZStream | |
| Target Release: | 13.0 (Queens) | Flags: | bhaley:
needinfo-
|
|
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-neutron-12.0.5-4.el7ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1707939 (view as bug list) | Environment: | ||
| Last Closed: | 2019-04-30 17:23:34 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1707939 | |||
So it looks like whenever we set the $device.forwarding sysctl we don't also set the all.forwarding sysctl. This is fine in the non-HA case where we set all.forwarding=1 at creation time and leave it alone. But in the HA case we start with all.forwarding=0, then set $device.forwarding=1 when the external device is added. So in short, it looks like that all.forwarding=1 call is missing in some circumstances, I'll work on tracking it down. Thanks for looking into this Brian. I also see the same behavior in OSP 10. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:0935 |
Description of problem: Note: this only impacts HA routers; non-HA routers function fine. With HA routers, the ipv6 kernel forwarding parameters are not being set correctly. Example: Active HA router: [root@overcloud-controller-0 ~]# ip netns exec qrouter-d21d169e-81c0-4218-9faa-61007d2fdd02 sysctl -a |grep ipv6|grep forward net.ipv6.conf.all.forwarding = 0 net.ipv6.conf.all.mc_forwarding = 0 net.ipv6.conf.default.forwarding = 0 net.ipv6.conf.default.mc_forwarding = 0 net.ipv6.conf.ha-1666cbe0-7f.forwarding = 0 net.ipv6.conf.ha-1666cbe0-7f.mc_forwarding = 0 net.ipv6.conf.lo.forwarding = 0 net.ipv6.conf.lo.mc_forwarding = 0 net.ipv6.conf.qg-a015ad18-fd.forwarding = 1 net.ipv6.conf.qg-a015ad18-fd.mc_forwarding = 0 net.ipv6.conf.qr-3a985db8-20.forwarding = 0 net.ipv6.conf.qr-3a985db8-20.mc_forwarding = 0 Non-HA router: # ip netns exec qrouter-56220321-e904-4834-bc2f-6567258f2799 sysctl -a |grep ipv6|grep forward net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.mc_forwarding = 0 net.ipv6.conf.default.forwarding = 1 net.ipv6.conf.default.mc_forwarding = 0 net.ipv6.conf.lo.forwarding = 1 net.ipv6.conf.lo.mc_forwarding = 0 net.ipv6.conf.qg-1e14ad34-57.forwarding = 1 net.ipv6.conf.qg-1e14ad34-57.mc_forwarding = 0 net.ipv6.conf.qr-3096a8c5-fe.forwarding = 1 net.ipv6.conf.qr-3096a8c5-fe.mc_forwarding = 0 Version-Release number of selected component (if applicable): Current OSP 13 python-neutron-12.0.3-5.el7ost.noarch How reproducible: 100% Steps to Reproduce: - OSP 13, default ML2/OVS with HA controllers $ neutron net-create test1 $ neutron subnet-create --name test1 --ip-version 6 --ipv6_address_mode=slaac --ipv6_ra_mode=slaac test1 2001:db8:ffff:ffff::/64 $ neutron router-create --ha true test1 $ neutron router-interface-add test1 test1 $ neutron router-gateway-set test1 external $ neutron l3-agent-list-hosting-router test1 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+-----------------------------------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+-----------------------------------+----------------+-------+----------+ | 74539ce6-52fe-4623-a328-11313ae9754c | overcloud-controller-0 | True | :-) | active | | 4533454a-1329-4733-95d7-415b3ec61a8a | overcloud-controller-2 | True | :-) | standby | | cde9066b-7033-46b6-90e7-af9208b0c648 | overcloud-controller-1 | True | :-) | standby | +--------------------------------------+-----------------------------------+----------------+-------+----------+ Set route for 2001:db8:ffff:ffff::/64 in upstream network if needed. $ nova boot --flavor rhel --image rhel1 --config-drive true --key-name default --nic net-id=`neutron net-show test1 |awk '$2 == "id"{print $4}'` test1 $ nova list +--------------------------------------+-------+--------+------------+-------------+----------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+----------------------------------------------+ | ef2c32de-96fe-49fc-b8af-6ddd05b9e499 | test1 | ACTIVE | - | Running | test1=2001:db8:ffff:ffff:f816:3eff:fe38:de0e | +--------------------------------------+-------+--------+------------+-------------+----------------------------------------------+ $ ping6 -c3 -W3 2001:db8:ffff:ffff:f816:3eff:fe38:de0e --- 2001:db8:ffff:ffff:f816:3eff:fe38:de0e ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 1999ms # inspect ipv6 forwarding rules in active router netns [root@overcloud-controller-0 ~]# ip netns exec qrouter-d21d169e-81c0-4218-9faa-61007d2fdd02 sysctl -a |grep ipv6|grep forward net.ipv6.conf.all.forwarding = 0 net.ipv6.conf.all.mc_forwarding = 0 net.ipv6.conf.default.forwarding = 0 net.ipv6.conf.default.mc_forwarding = 0 net.ipv6.conf.ha-1666cbe0-7f.forwarding = 0 net.ipv6.conf.ha-1666cbe0-7f.mc_forwarding = 0 net.ipv6.conf.lo.forwarding = 0 net.ipv6.conf.lo.mc_forwarding = 0 net.ipv6.conf.qg-a015ad18-fd.forwarding = 1 net.ipv6.conf.qg-a015ad18-fd.mc_forwarding = 0 net.ipv6.conf.qr-3a985db8-20.forwarding = 0 net.ipv6.conf.qr-3a985db8-20.mc_forwarding = 0 # manually enable net.ipv6.conf.all.forwarding [root@overcloud-controller-0 ~]# ip netns exec qrouter-d21d169e-81c0-4218-9faa-61007d2fdd02 sysctl net.ipv6.conf.all.forwarding=1 net.ipv6.conf.all.forwarding = 1 # Ping now works. $ ping6 -c3 -W3 2001:db8:ffff:ffff:f816:3eff:fe38:de0e PING 2001:db8:ffff:ffff:f816:3eff:fe38:de0e 56 data bytes 64 bytes from 2001:db8:ffff:ffff:f816:3eff:fe38:de0e: icmp_seq=1 ttl=62 time=4.16 ms 64 bytes from 2001:db8:ffff:ffff:f816:3eff:fe38:de0e: icmp_seq=2 ttl=62 time=1.61 ms 64 bytes from 2001:db8:ffff:ffff:f816:3eff:fe38:de0e: icmp_seq=3 ttl=62 time=1.77 ms --- 2001:db8:ffff:ffff:f816:3eff:fe38:de0e ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 1.610/2.516/4.165/1.167 ms