Bug 1287809 - During 7.1 -> 7.2 update 2 of the 3 neutron l3 agents show in active ha_state
During 7.1 -> 7.2 update 2 of the 3 neutron l3 agents show in active ha_state
Status: CLOSED DUPLICATE of bug 1260298
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity high
: ---
: 8.0 (Liberty)
Assigned To: chris alfonso
yeylon@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-02 13:11 EST by Marius Cornea
Modified: 2016-04-18 02:52 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-03 06:06:49 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
updates l3 check (3.60 KB, text/plain)
2015-12-02 13:11 EST, Marius Cornea
no flags Details

  None (edit)
Description Marius Cornea 2015-12-02 13:11:08 EST
Created attachment 1101573 [details]
updates l3 check

Description of problem:
During 7.1 -> 7.2 update 2 of the 3 neutron l3 agents shows in active ha_state.
This shows up after yum update has been run on all the controllers. This, however, doesn't seem to interrupt floating IPs connectivity as the IP addresses are set up only on one of the namespaces. 

Version-Release number of selected component (if applicable):
openstack-neutron-openvswitch-2015.1.2-2.el7ost.noarch
openstack-neutron-2015.1.2-2.el7ost.noarch
python-neutron-2015.1.2-2.el7ost.noarch
openstack-neutron-ml2-2015.1.2-2.el7ost.noarch
openstack-neutron-common-2015.1.2-2.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy 7.1 overcloud with 3 ctrls 
2. Set up router with external network 
3. Update to 7.2 

Actual results:
When running neutron l3-agent-list-hosting-router tenant-router 2 of the l3 agents show up as active in the ha_state column.

Expected results:
Only one agent shows as active ha_state. 

Additional info:
Attaching the commands output.
Comment 2 Jiri Stransky 2015-12-03 06:06:49 EST
I investigated the environment and the root cause seems to be bug 1260298. The tenant router on the environment is in active state only on controller-1, but neutron reports it incorrectly also on controller-2. The real allocation of the router can be queried by inspecting the router network namespace, e.g. by running `ip netns exec $(ip netns | grep qrouter) ip a` on the controller and seeing where the router & instance floating IPs are allocated. (Such command only works if you have just one qrouter namespace on the controller.)

stack@instack:~>>> neutron l3-agent-list-hosting-router tenant-router
+--------------------------------------+------------------------------------+----------------+-------+----------+
| id                                   | host                               | admin_state_up | alive | ha_state |
+--------------------------------------+------------------------------------+----------------+-------+----------+
| 36a63523-bd2d-42d3-814e-21d08e4cd8b9 | overcloud-controller-1.localdomain | True           | :-)   | active   |
| fd484ddb-d89f-4e64-867a-eadb8727f474 | overcloud-controller-2.localdomain | True           | :-)   | active   |
| 455d6624-7039-49b2-87fd-44d23cc2924b | overcloud-controller-0.localdomain | True           | :-)   | standby  |
+--------------------------------------+------------------------------------+----------------+-------+----------+

^ router reported on c-1 and c-2, but in fact it's only on c-1:

[root@overcloud-controller-0 ~]# ip netns exec $(ip netns | grep qrouter) ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
17: ha-e91c2c33-88: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:51:1c:ae brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-e91c2c33-88
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe51:1cae/64 scope link 
       valid_lft forever preferred_lft forever
18: qr-9a9e55e2-cc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:b7:60:6c brd ff:ff:ff:ff:ff:ff
19: qg-9ae0de5a-9b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:6d:96:05 brd ff:ff:ff:ff:ff:ff

[root@overcloud-controller-1 ~]# ip netns exec $(ip netns | grep qrouter) ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
17: ha-13f9b1eb-a2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:8a:e0:ea brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.3/18 brd 169.254.255.255 scope global ha-13f9b1eb-a2
       valid_lft forever preferred_lft forever
    inet 169.254.0.1/24 scope global ha-13f9b1eb-a2
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe8a:e0ea/64 scope link 
       valid_lft forever preferred_lft forever
18: qr-9a9e55e2-cc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:b7:60:6c brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 scope global qr-9a9e55e2-cc
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feb7:606c/64 scope link nodad 
       valid_lft forever preferred_lft forever
19: qg-9ae0de5a-9b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:6d:96:05 brd ff:ff:ff:ff:ff:ff
    inet 172.16.23.110/24 scope global qg-9ae0de5a-9b
       valid_lft forever preferred_lft forever
    inet 172.16.23.111/32 scope global qg-9ae0de5a-9b
       valid_lft forever preferred_lft forever
    inet 172.16.23.112/32 scope global qg-9ae0de5a-9b
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe6d:9605/64 scope link nodad 
       valid_lft forever preferred_lft forever

[root@overcloud-controller-2 ~]# ip netns exec $(ip netns | grep qrouter) ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
13: ha-2c657e2a-5e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:fa:a3:a3 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-2c657e2a-5e
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fefa:a3a3/64 scope link 
       valid_lft forever preferred_lft forever
14: qr-9a9e55e2-cc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:b7:60:6c brd ff:ff:ff:ff:ff:ff
15: qg-9ae0de5a-9b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:6d:96:05 brd ff:ff:ff:ff:ff:ff

I'll close it as a duplicate of the root cause bug, please re-open if deemed  necessary.

*** This bug has been marked as a duplicate of bug 1260298 ***

Note You need to log in before you can comment on or make changes to this bug.