Bug 1287809
| Summary: | During 7.1 -> 7.2 update 2 of the 3 neutron l3 agents show in active ha_state | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marius Cornea <mcornea> | ||||
| Component: | rhosp-director | Assignee: | chris alfonso <calfonso> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | yeylon <yeylon> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.0 (Kilo) | CC: | hbrock, jstransk, mburns, rhel-osp-director-maint, srevivo | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 8.0 (Liberty) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2015-12-03 11:06:49 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
I investigated the environment and the root cause seems to be bug 1260298. The tenant router on the environment is in active state only on controller-1, but neutron reports it incorrectly also on controller-2. The real allocation of the router can be queried by inspecting the router network namespace, e.g. by running `ip netns exec $(ip netns | grep qrouter) ip a` on the controller and seeing where the router & instance floating IPs are allocated. (Such command only works if you have just one qrouter namespace on the controller.) stack@instack:~>>> neutron l3-agent-list-hosting-router tenant-router +--------------------------------------+------------------------------------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+------------------------------------+----------------+-------+----------+ | 36a63523-bd2d-42d3-814e-21d08e4cd8b9 | overcloud-controller-1.localdomain | True | :-) | active | | fd484ddb-d89f-4e64-867a-eadb8727f474 | overcloud-controller-2.localdomain | True | :-) | active | | 455d6624-7039-49b2-87fd-44d23cc2924b | overcloud-controller-0.localdomain | True | :-) | standby | +--------------------------------------+------------------------------------+----------------+-------+----------+ ^ router reported on c-1 and c-2, but in fact it's only on c-1: [root@overcloud-controller-0 ~]# ip netns exec $(ip netns | grep qrouter) ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 17: ha-e91c2c33-88: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:51:1c:ae brd ff:ff:ff:ff:ff:ff inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-e91c2c33-88 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe51:1cae/64 scope link valid_lft forever preferred_lft forever 18: qr-9a9e55e2-cc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:b7:60:6c brd ff:ff:ff:ff:ff:ff 19: qg-9ae0de5a-9b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:6d:96:05 brd ff:ff:ff:ff:ff:ff [root@overcloud-controller-1 ~]# ip netns exec $(ip netns | grep qrouter) ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 17: ha-13f9b1eb-a2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:8a:e0:ea brd ff:ff:ff:ff:ff:ff inet 169.254.192.3/18 brd 169.254.255.255 scope global ha-13f9b1eb-a2 valid_lft forever preferred_lft forever inet 169.254.0.1/24 scope global ha-13f9b1eb-a2 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe8a:e0ea/64 scope link valid_lft forever preferred_lft forever 18: qr-9a9e55e2-cc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:b7:60:6c brd ff:ff:ff:ff:ff:ff inet 192.168.0.1/24 scope global qr-9a9e55e2-cc valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feb7:606c/64 scope link nodad valid_lft forever preferred_lft forever 19: qg-9ae0de5a-9b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:6d:96:05 brd ff:ff:ff:ff:ff:ff inet 172.16.23.110/24 scope global qg-9ae0de5a-9b valid_lft forever preferred_lft forever inet 172.16.23.111/32 scope global qg-9ae0de5a-9b valid_lft forever preferred_lft forever inet 172.16.23.112/32 scope global qg-9ae0de5a-9b valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe6d:9605/64 scope link nodad valid_lft forever preferred_lft forever [root@overcloud-controller-2 ~]# ip netns exec $(ip netns | grep qrouter) ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 13: ha-2c657e2a-5e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:fa:a3:a3 brd ff:ff:ff:ff:ff:ff inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-2c657e2a-5e valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fefa:a3a3/64 scope link valid_lft forever preferred_lft forever 14: qr-9a9e55e2-cc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:b7:60:6c brd ff:ff:ff:ff:ff:ff 15: qg-9ae0de5a-9b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:6d:96:05 brd ff:ff:ff:ff:ff:ff I'll close it as a duplicate of the root cause bug, please re-open if deemed necessary. *** This bug has been marked as a duplicate of bug 1260298 *** |
Created attachment 1101573 [details] updates l3 check Description of problem: During 7.1 -> 7.2 update 2 of the 3 neutron l3 agents shows in active ha_state. This shows up after yum update has been run on all the controllers. This, however, doesn't seem to interrupt floating IPs connectivity as the IP addresses are set up only on one of the namespaces. Version-Release number of selected component (if applicable): openstack-neutron-openvswitch-2015.1.2-2.el7ost.noarch openstack-neutron-2015.1.2-2.el7ost.noarch python-neutron-2015.1.2-2.el7ost.noarch openstack-neutron-ml2-2015.1.2-2.el7ost.noarch openstack-neutron-common-2015.1.2-2.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy 7.1 overcloud with 3 ctrls 2. Set up router with external network 3. Update to 7.2 Actual results: When running neutron l3-agent-list-hosting-router tenant-router 2 of the l3 agents show up as active in the ha_state column. Expected results: Only one agent shows as active ha_state. Additional info: Attaching the commands output.