Bug 1727350
| Summary: | [OSP15][L3 ha]When reboot a controller with the router master role, during sometime several network nodes are shown as master | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Candido Campos <ccamposr> |
| Component: | openstack-neutron | Assignee: | Rodolfo Alonso <ralonsoh> |
| Status: | CLOSED ERRATA | QA Contact: | Eran Kuris <ekuris> |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 15.0 (Stein) | CC: | amuller, chrisw, ekuris, njohnston, scohen |
| Target Milestone: | --- | Keywords: | Triaged, ZStream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-neutron-14.0.3-0.20190912080435.d63adef.el8ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-03-05 11:53:49 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0709 |
Description of problem: When reboot a controller with the router master role, during sometime several network nodes are shown as maste Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.Deploy a router with a L3 ha 2.reboot the controller with master role 3. Actual results: During some time several networker are shown as master and several pings are lost Expected results: No ping lost or 1 and only one networker shown as master Additional info: (overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+--------------------------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+--------------------------+----------------+-------+----------+ | 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True | :-) | standby | | 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True | :-) | standby | | edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True | :-) | active | +--------------------------------------+--------------------------+----------------+-------+----------+ (overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+--------------------------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+--------------------------+----------------+-------+----------+ | 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True | :-) | active | | 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True | :-) | active | | edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True | :-) | active | +--------------------------------------+--------------------------+----------------+-------+----------+ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+--------------------------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+--------------------------+----------------+-------+----------+ | 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True | :-) | active | | 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True | :-) | active | | edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True | :-) | active | +--------------------------------------+--------------------------+----------------+-------+----------+ (overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+--------------------------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+--------------------------+----------------+-------+----------+ | 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True | :-) | active | | 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True | :-) | active | | edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True | :-) | active | +--------------------------------------+--------------------------+----------------+-------+----------+ (overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+--------------------------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+--------------------------+----------------+-------+----------+ | 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True | :-) | active | | 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True | :-) | active | | edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True | :-) | standby | +--------------------------------------+--------------------------+----------------+-------+----------+ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+--------------------------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+--------------------------+----------------+-------+----------+ | 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True | :-) | active | | 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True | :-) | active | | edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True | :-) | standby | +--------------------------------------+--------------------------+----------------+-------+----------+ (overcloud) [stack@undercloud-0 ~]$ openstack floating ip create public after a secon reboot: (overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+--------------------------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+--------------------------+----------------+-------+----------+ | 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True | :-) | active | | 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True | :-) | standby | | edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True | :-) | active | +--------------------------------------+--------------------------+----------------+-------+----------+ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+--------------------------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+--------------------------+----------------+-------+----------+ | 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True | :-) | active | | 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True | :-) | standby | | edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True | :-) | active | +--------------------------------------+--------------------------+----------------+-------+----------+ In the networkers only one is master: [root@controller-0 heat-admin]# cat /var/lib/neutron/ha_confs/*/state [root@controller-0 heat-admin]# ip netns exec qrouter-e13f0bcf-52d5-4db9-bec6-59e1058244e1 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 16: ha-3ebb3167-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:85:84:80 brd ff:ff:ff:ff:ff:ff inet 169.254.193.153/18 brd 169.254.255.255 scope global ha-3ebb3167-a0 valid_lft forever preferred_lft forever inet 169.254.0.30/24 scope global ha-3ebb3167-a0 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe85:8480/64 scope link valid_lft forever preferred_lft forever 17: qr-875cb1f8-85: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:10:e9:dc brd ff:ff:ff:ff:ff:ff inet 10.1.0.1/24 scope global qr-875cb1f8-85 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe10:e9dc/64 scope link nodad valid_lft forever preferred_lft forever 18: qg-0df89c2a-ad: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:d5:31:e1 brd ff:ff:ff:ff:ff:ff inet 10.0.0.215/24 scope global qg-0df89c2a-ad valid_lft forever preferred_lft forever inet 10.0.0.241/32 scope global qg-0df89c2a-ad valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fed5:31e1/64 scope link nodad valid_lft forever preferred_lft forever [root@controller-0 heat-admin]# [root@controller-0 heat-admin]# cat /var/lib/neutron/ha_confs/*/state [root@controller-0 heat-admin]# ip netns exec qrouter-e13f0bcf-52d5-4db9-bec6-59e1058244e1 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 16: ha-3ebb3167-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:85:84:80 brd ff:ff:ff:ff:ff:ff inet 169.254.193.153/18 brd 169.254.255.255 scope global ha-3ebb3167-a0 valid_lft forever preferred_lft forever inet 169.254.0.30/24 scope global ha-3ebb3167-a0 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe85:8480/64 scope link valid_lft forever preferred_lft forever 17: qr-875cb1f8-85: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:10:e9:dc brd ff:ff:ff:ff:ff:ff inet 10.1.0.1/24 scope global qr-875cb1f8-85 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe10:e9dc/64 scope link nodad valid_lft forever preferred_lft forever 18: qg-0df89c2a-ad: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:d5:31:e1 brd ff:ff:ff:ff:ff:ff inet 10.0.0.215/24 scope global qg-0df89c2a-ad valid_lft forever preferred_lft forever inet 10.0.0.241/32 scope global qg-0df89c2a-ad valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fed5:31e1/64 scope link nodad valid_lft forever preferred_lft forever [root@controller-0 heat-admin]# cat /var/lib/neutron/ha_confs/*/state backup[root@controller-2 heat-admin]# ip netns exec qrouter-e13f0bcf-52d5-4db9-bec6-59e1058244e1 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 20: ha-b63fa291-b8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:95:1e:e3 brd ff:ff:ff:ff:ff:ff inet 169.254.193.239/18 brd 169.254.255.255 scope global ha-b63fa291-b8 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe95:1ee3/64 scope link valid_lft forever preferred_lft forever 23: qr-875cb1f8-85: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:10:e9:dc brd ff:ff:ff:ff:ff:ff 24: qg-0df89c2a-ad: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:d5:31:e1 brd ff:ff:ff:ff:ff:ff [root@controller-2 heat-admin]# (undercloud) [stack@undercloud-0 ~]$ ping 10.0.0.241 PING 10.0.0.241 (10.0.0.241) 56(84) bytes of data. 64 bytes from 10.0.0.241: icmp_seq=1 ttl=63 time=2.54 ms 64 bytes from 10.0.0.241: icmp_seq=2 ttl=63 time=1.98 ms 64 bytes from 10.0.0.241: icmp_seq=3 ttl=63 time=1.02 ms ^C --- 10.0.0.241 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 4ms rtt min/avg/max/mdev = 1.021/1.844/2.535/0.626 ms (undercloud) [stack@undercloud-0 ~]$ ping 10.0.0.241 PING 10.0.0.241 (10.0.0.241) 56(84) bytes of data. 64 bytes from 10.0.0.241: icmp_seq=1 ttl=63 time=1.44 ms 64 bytes from 10.0.0.241: icmp_seq=2 ttl=63 time=1.25 ms 64 bytes from 10.0.0.241: icmp_seq=3 ttl=63 time=1.19 ms 64 bytes from 10.0.0.241: icmp_seq=4 ttl=63 time=1.11 ms 64 bytes from 10.0.0.241: icmp_seq=5 ttl=63 time=1.04 ms 64 bytes from 10.0.0.241: icmp_seq=6 ttl=63 time=1.07 ms 64 bytes from 10.0.0.241: icmp_seq=7 ttl=63 time=1.25 ms 64 bytes from 10.0.0.241: icmp_seq=8 ttl=63 time=1.18 ms 64 bytes from 10.0.0.241: icmp_seq=9 ttl=63 time=0.903 ms 64 bytes from 10.0.0.241: icmp_seq=10 ttl=63 time=1.04 ms 64 bytes from 10.0.0.241: icmp_seq=11 ttl=63 time=1.07 ms 64 bytes from 10.0.0.241: icmp_seq=16 ttl=63 time=2.69 ms 64 bytes from 10.0.0.241: icmp_seq=17 ttl=63 time=1.01 ms 64 bytes from 10.0.0.241: icmp_seq=18 ttl=63 time=1.07 ms 64 bytes from 10.0.0.241: icmp_seq=19 ttl=63 time=1.87 ms 64 bytes from 10.0.0.241: icmp_seq=20 ttl=63 time=1.15 ms 64 bytes from 10.0.0.241: icmp_seq=21 ttl=63 time=1.05 ms 64 bytes from 10.0.0.241: icmp_seq=22 ttl=63 time=1.30 ms 64 bytes from 10.0.0.241: icmp_seq=23 ttl=63 time=1.02 ms 64 bytes from 10.0.0.241: icmp_seq=24 ttl=63 time=0.915 ms 64 bytes from 10.0.0.241: icmp_seq=25 ttl=63 time=0.941 ms 64 bytes from 10.0.0.241: icmp_seq=26 ttl=63 time=1.17 ms 64 bytes from 10.0.0.241: icmp_seq=27 ttl=63 time=1.32 ms 64 bytes from 10.0.0.241: icmp_seq=28 ttl=63 time=1.07 ms 64 bytes from 10.0.0.241: icmp_seq=29 ttl=63 time=1.38 ms 64 bytes from 10.0.0.241: icmp_seq=30 ttl=63 time=1.25 ms ^C --- 10.0.0.241 ping statistics --- 30 packets transmitted, 26 received, 13.3333% packet loss, time 148ms rtt min/avg/max/mdev = 0.903/1.220/2.688/0.355 ms (undercloud) [stack@undercloud-0 ~]$ 64 bytes from 10.0.0.241: icmp_seq=130 ttl=63 time=0.978 ms 64 bytes from 10.0.0.241: icmp_seq=131 ttl=63 time=1.15 ms 64 bytes from 10.0.0.241: icmp_seq=132 ttl=63 time=1.07 ms ^C --- 10.0.0.241 ping statistics --- 132 packets transmitted, 125 received, 5.30303% packet loss, time 897ms rtt min/avg/max/mdev = 0.690/1.191/3.666/0.405 ms (undercloud) [stack@undercloud-0 ~]$