Bug 1727350 - [OSP15][L3 ha]When reboot a controller with the router master role, during sometime several network nodes are shown as master
Summary: [OSP15][L3 ha]When reboot a controller with the router master role, during so...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Rodolfo Alonso
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-05 15:53 UTC by Candido Campos
Modified: 2020-03-05 11:54 UTC (History)
5 users (show)

Fixed In Version: openstack-neutron-14.0.3-0.20190912080435.d63adef.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-03-05 11:53:49 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1837635 0 None None None 2019-07-24 14:02:59 UTC
OpenStack gerrit 679431 0 None MERGED Refactor the L3 agent batch notifier 2020-09-21 12:37:29 UTC
OpenStack gerrit 679438 0 None MERGED Delay HA router transition from "backup" to "master" 2020-09-21 12:37:29 UTC
Red Hat Product Errata RHBA-2020:0709 0 None None None 2020-03-05 11:54:19 UTC

Description Candido Campos 2019-07-05 15:53:05 UTC
Description of problem:

When reboot a controller with the router master role, during sometime several network nodes are shown as maste

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.Deploy a router with a L3 ha
2.reboot the controller with master role
3.

Actual results:

During some time several networker are shown as master and several pings are lost

Expected results:

No ping lost or 1 and only one networker shown as master

Additional info:

(overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------------+----------------+-------+----------+
| id                                   | host                     | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True           | :-)   | standby  |
| 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True           | :-)   | standby  |
| edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True           | :-)   | active   |
+--------------------------------------+--------------------------+----------------+-------+----------+
(overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------------+----------------+-------+----------+
| id                                   | host                     | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True           | :-)   | active   |
| 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True           | :-)   | active   |
| edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True           | :-)   | active   |
+--------------------------------------+--------------------------+----------------+-------+----------+
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------------+----------------+-------+----------+
| id                                   | host                     | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True           | :-)   | active   |
| 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True           | :-)   | active   |
| edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True           | :-)   | active   |
+--------------------------------------+--------------------------+----------------+-------+----------+
(overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------------+----------------+-------+----------+
| id                                   | host                     | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True           | :-)   | active   |
| 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True           | :-)   | active   |
| edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True           | :-)   | active   |
+--------------------------------------+--------------------------+----------------+-------+----------+
(overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------------+----------------+-------+----------+
| id                                   | host                     | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True           | :-)   | active   |
| 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True           | :-)   | active   |
| edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True           | :-)   | standby  |
+--------------------------------------+--------------------------+----------------+-------+----------+
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------------+----------------+-------+----------+
| id                                   | host                     | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True           | :-)   | active   |
| 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True           | :-)   | active   |
| edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True           | :-)   | standby  |
+--------------------------------------+--------------------------+----------------+-------+----------+
(overcloud) [stack@undercloud-0 ~]$ openstack floating ip create public


after a secon reboot:


(overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------------+----------------+-------+----------+
| id                                   | host                     | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True           | :-)   | active   |
| 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True           | :-)   | standby  |
| edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True           | :-)   | active   |
+--------------------------------------+--------------------------+----------------+-------+----------+
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router router
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------------+----------------+-------+----------+
| id                                   | host                     | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| 4056cd8e-e062-4f45-bc83-d3eb51905ff5 | controller-0.localdomain | True           | :-)   | active   |
| 527d6a6c-8d2e-4796-bbd0-8b41cf365743 | controller-2.localdomain | True           | :-)   | standby  |
| edbdfc1c-3505-4891-8d00-f3a6308bb1de | controller-1.localdomain | True           | :-)   | active   |
+--------------------------------------+--------------------------+----------------+-------+----------+
 In the networkers only one is master:

[root@controller-0 heat-admin]# cat /var/lib/neutron/ha_confs/*/state
[root@controller-0 heat-admin]# ip netns exec qrouter-e13f0bcf-52d5-4db9-bec6-59e1058244e1 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
16: ha-3ebb3167-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:85:84:80 brd ff:ff:ff:ff:ff:ff
    inet 169.254.193.153/18 brd 169.254.255.255 scope global ha-3ebb3167-a0
       valid_lft forever preferred_lft forever
    inet 169.254.0.30/24 scope global ha-3ebb3167-a0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe85:8480/64 scope link 
       valid_lft forever preferred_lft forever
17: qr-875cb1f8-85: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:10:e9:dc brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/24 scope global qr-875cb1f8-85
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe10:e9dc/64 scope link nodad 
       valid_lft forever preferred_lft forever
18: qg-0df89c2a-ad: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:d5:31:e1 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.215/24 scope global qg-0df89c2a-ad
       valid_lft forever preferred_lft forever
    inet 10.0.0.241/32 scope global qg-0df89c2a-ad
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fed5:31e1/64 scope link nodad 
       valid_lft forever preferred_lft forever
[root@controller-0 heat-admin]# 

[root@controller-0 heat-admin]# cat /var/lib/neutron/ha_confs/*/state
[root@controller-0 heat-admin]# ip netns exec qrouter-e13f0bcf-52d5-4db9-bec6-59e1058244e1 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
16: ha-3ebb3167-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:85:84:80 brd ff:ff:ff:ff:ff:ff
    inet 169.254.193.153/18 brd 169.254.255.255 scope global ha-3ebb3167-a0
       valid_lft forever preferred_lft forever
    inet 169.254.0.30/24 scope global ha-3ebb3167-a0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe85:8480/64 scope link 
       valid_lft forever preferred_lft forever
17: qr-875cb1f8-85: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:10:e9:dc brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/24 scope global qr-875cb1f8-85
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe10:e9dc/64 scope link nodad 
       valid_lft forever preferred_lft forever
18: qg-0df89c2a-ad: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:d5:31:e1 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.215/24 scope global qg-0df89c2a-ad
       valid_lft forever preferred_lft forever
    inet 10.0.0.241/32 scope global qg-0df89c2a-ad
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fed5:31e1/64 scope link nodad 
       valid_lft forever preferred_lft forever
[root@controller-0 heat-admin]# 

cat /var/lib/neutron/ha_confs/*/state
backup[root@controller-2 heat-admin]# ip netns exec qrouter-e13f0bcf-52d5-4db9-bec6-59e1058244e1 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: ha-b63fa291-b8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:95:1e:e3 brd ff:ff:ff:ff:ff:ff
    inet 169.254.193.239/18 brd 169.254.255.255 scope global ha-b63fa291-b8
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe95:1ee3/64 scope link 
       valid_lft forever preferred_lft forever
23: qr-875cb1f8-85: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:10:e9:dc brd ff:ff:ff:ff:ff:ff
24: qg-0df89c2a-ad: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:d5:31:e1 brd ff:ff:ff:ff:ff:ff
[root@controller-2 heat-admin]# 



(undercloud) [stack@undercloud-0 ~]$ ping 10.0.0.241
PING 10.0.0.241 (10.0.0.241) 56(84) bytes of data.
64 bytes from 10.0.0.241: icmp_seq=1 ttl=63 time=2.54 ms
64 bytes from 10.0.0.241: icmp_seq=2 ttl=63 time=1.98 ms
64 bytes from 10.0.0.241: icmp_seq=3 ttl=63 time=1.02 ms
^C
--- 10.0.0.241 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 1.021/1.844/2.535/0.626 ms
(undercloud) [stack@undercloud-0 ~]$ ping 10.0.0.241
PING 10.0.0.241 (10.0.0.241) 56(84) bytes of data.
64 bytes from 10.0.0.241: icmp_seq=1 ttl=63 time=1.44 ms
64 bytes from 10.0.0.241: icmp_seq=2 ttl=63 time=1.25 ms
64 bytes from 10.0.0.241: icmp_seq=3 ttl=63 time=1.19 ms
64 bytes from 10.0.0.241: icmp_seq=4 ttl=63 time=1.11 ms
64 bytes from 10.0.0.241: icmp_seq=5 ttl=63 time=1.04 ms
64 bytes from 10.0.0.241: icmp_seq=6 ttl=63 time=1.07 ms
64 bytes from 10.0.0.241: icmp_seq=7 ttl=63 time=1.25 ms
64 bytes from 10.0.0.241: icmp_seq=8 ttl=63 time=1.18 ms
64 bytes from 10.0.0.241: icmp_seq=9 ttl=63 time=0.903 ms
64 bytes from 10.0.0.241: icmp_seq=10 ttl=63 time=1.04 ms
64 bytes from 10.0.0.241: icmp_seq=11 ttl=63 time=1.07 ms
64 bytes from 10.0.0.241: icmp_seq=16 ttl=63 time=2.69 ms
64 bytes from 10.0.0.241: icmp_seq=17 ttl=63 time=1.01 ms
64 bytes from 10.0.0.241: icmp_seq=18 ttl=63 time=1.07 ms
64 bytes from 10.0.0.241: icmp_seq=19 ttl=63 time=1.87 ms
64 bytes from 10.0.0.241: icmp_seq=20 ttl=63 time=1.15 ms
64 bytes from 10.0.0.241: icmp_seq=21 ttl=63 time=1.05 ms
64 bytes from 10.0.0.241: icmp_seq=22 ttl=63 time=1.30 ms
64 bytes from 10.0.0.241: icmp_seq=23 ttl=63 time=1.02 ms
64 bytes from 10.0.0.241: icmp_seq=24 ttl=63 time=0.915 ms
64 bytes from 10.0.0.241: icmp_seq=25 ttl=63 time=0.941 ms
64 bytes from 10.0.0.241: icmp_seq=26 ttl=63 time=1.17 ms
64 bytes from 10.0.0.241: icmp_seq=27 ttl=63 time=1.32 ms
64 bytes from 10.0.0.241: icmp_seq=28 ttl=63 time=1.07 ms
64 bytes from 10.0.0.241: icmp_seq=29 ttl=63 time=1.38 ms
64 bytes from 10.0.0.241: icmp_seq=30 ttl=63 time=1.25 ms
^C
--- 10.0.0.241 ping statistics ---
30 packets transmitted, 26 received, 13.3333% packet loss, time 148ms
rtt min/avg/max/mdev = 0.903/1.220/2.688/0.355 ms
(undercloud) [stack@undercloud-0 ~]$ 


64 bytes from 10.0.0.241: icmp_seq=130 ttl=63 time=0.978 ms
64 bytes from 10.0.0.241: icmp_seq=131 ttl=63 time=1.15 ms
64 bytes from 10.0.0.241: icmp_seq=132 ttl=63 time=1.07 ms
^C
--- 10.0.0.241 ping statistics ---
132 packets transmitted, 125 received, 5.30303% packet loss, time 897ms
rtt min/avg/max/mdev = 0.690/1.191/3.666/0.405 ms
(undercloud) [stack@undercloud-0 ~]$

Comment 9 errata-xmlrpc 2020-03-05 11:53:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0709


Note You need to log in before you can comment on or make changes to this bug.