Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1829757

Summary: Stale neutron-keepalived-qrouter containers after the router delete in OVS deployed RHSOP-16
Product: Red Hat OpenStack Reporter: Srinivas Atmakuri <satmakur>
Component: openstack-neutronAssignee: Assaf Muller <amuller>
Status: CLOSED DUPLICATE QA Contact: Eran Kuris <ekuris>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 16.0 (Train)CC: amuller, bcafarel, bperkins, chrisw, scohen
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-30 13:56:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Srinivas Atmakuri 2020-04-30 09:38:21 UTC 
Description of problem:
Stale neutron-keepalived-qrouter containers after the router delete in OVS deployed RHSOP-16

Version-Release number of selected component (if applicable):
Red Hat Openstack Platform - 16 (RHOSP-16)

How reproducible:
Always

Steps to Reproduce:
1. Create a router
2. Attach a gateway and detach
3. Now, delete the router

Actual results:
neutron-keepalived-qrouter-$ID present in UP state on the controllers nodes

Expected results:
neutron-keepalived-qrouter-$ID containers also deleted with router delete

Additional info:
(overcloud) [stack@machine-16 ~]$ openstack network list
+--------------------------------------+----------------------------------------------------+--------------------------------------+
| ID                                   | Name                                               | Subnets                              |
+--------------------------------------+----------------------------------------------------+--------------------------------------+
| 09881516-cd27-45bb-8eeb-25de315fe8ee | HA network tenant 671470eaddec445db420e9cf5bddd969 | cee2c86e-2aab-46e0-940a-6e4aa60df233 |
| 8a0de287-0c31-494d-b221-72752914b266 | testnetwork                                        | 8826a9f1-4beb-4bef-bac4-bae29037090f |
+--------------------------------------+----------------------------------------------------+--------------------------------------+
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ openstack subnet list
+--------------------------------------+---------------------------------------------------+--------------------------------------+------------------+
| ID                                   | Name                                              | Network                              | Subnet           |
+--------------------------------------+---------------------------------------------------+--------------------------------------+------------------+
| 8826a9f1-4beb-4bef-bac4-bae29037090f | testsubnet                                        | 8a0de287-0c31-494d-b221-72752914b266 | 192.168.122.0/24 |
| cee2c86e-2aab-46e0-940a-6e4aa60df233 | HA subnet tenant 671470eaddec445db420e9cf5bddd969 | 09881516-cd27-45bb-8eeb-25de315fe8ee | 169.254.192.0/18 |
+--------------------------------------+---------------------------------------------------+--------------------------------------+------------------+

(overcloud) [stack@machine-16 ~]$ openstack router list
+--------------------------------------+-------+--------+-------+----------------------------------+------+
| ID                                   | Name  | Status | State | Project                          | HA   |
+--------------------------------------+-------+--------+-------+----------------------------------+------+
| 253b1c97-4105-4bf6-9346-374b34fb92ad | route | ACTIVE | UP    | 671470eaddec445db420e9cf5bddd969 | True |
+--------------------------------------+-------+--------+-------+----------------------------------+------+
(overcloud) [stack@machine-16 ~]$


(undercloud) [stack@machine-16 ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks               |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| 6b6cce33-c2a8-43be-9cd5-fbce94c1add0 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.168.24.11 |
| 9220f843-dd8c-41d5-a1c9-ad88d148ead7 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.168.24.15 |
| 7fad735e-076a-4e29-a6e6-251b7fea79ba | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.168.24.21 |
| 55c88777-19bb-4402-9a09-b6599b6d328c | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.7  |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
(undercloud) [stack@machine-16 ~]$


(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.11 "sudo podman ps | grep -i neutron"
ee70a6a377a5  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           /usr/sbin/keepali...  7 minutes ago      Up 7 minutes ago             neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad
b6715a07ab99  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         /usr/sbin/dnsmasq...  About an hour ago  Up About an hour ago         neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266
f33d4da542f3  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           About an hour ago  Up About an hour ago         neutron_ovs_agent
9621b3fe1898  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           About an hour ago  Up About an hour ago         neutron_metadata_agent
ea13d04586a9  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           kolla_start           About an hour ago  Up About an hour ago         neutron_l3_agent
f02c8d307e48  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         kolla_start           About an hour ago  Up About an hour ago         neutron_dhcp
11a933635eed  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           2 hours ago        Up 2 hours ago               neutron_api
(overcloud) [stack@machine-16 ~]$ 


(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.15 "sudo podman ps | grep -i neutron"
515d7b57d109  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           /bin/bash -c HAPR...  7 minutes ago      Up 7 minutes ago             neutron-haproxy-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad
edddc818df24  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           /usr/sbin/keepali...  7 minutes ago      Up 7 minutes ago             neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad
736f825342b3  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         /usr/sbin/dnsmasq...  About an hour ago  Up About an hour ago         neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266
3a55cce49e0b  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           About an hour ago  Up About an hour ago         neutron_ovs_agent
55258961cb36  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           About an hour ago  Up About an hour ago         neutron_metadata_agent
0ec4ddf9aa39  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           kolla_start           About an hour ago  Up About an hour ago         neutron_l3_agent
00e3e24c5107  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         kolla_start           About an hour ago  Up About an hour ago         neutron_dhcp
c469edaae51a  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           2 hours ago        Up 2 hours ago               neutron_api
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 


(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.21 "sudo podman ps | grep -i neutron"
922358f8cc13  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           /usr/sbin/keepali...  7 minutes ago      Up 7 minutes ago             neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad
eaedd62297a4  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         /usr/sbin/dnsmasq...  About an hour ago  Up About an hour ago         neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266
f6c508d26299  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           About an hour ago  Up About an hour ago         neutron_ovs_agent
1a3a1498ea85  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           About an hour ago  Up About an hour ago         neutron_metadata_agent
955e6c47dd80  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           kolla_start           About an hour ago  Up About an hour ago         neutron_l3_agent
2699e7b00475  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         kolla_start           About an hour ago  Up About an hour ago         neutron_dhcp
4d892b53d526  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           2 hours ago        Up 2 hours ago               neutron_api



openstack router unset --external-gateway route

(overcloud) [stack@machine-16 ~]$ openstack router delete route
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ openstack router list

(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.21 "sudo podman ps | grep -i neutron"
922358f8cc13  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           /usr/sbin/keepali...  14 minutes ago     Up 14 minutes ago            neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad
eaedd62297a4  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         /usr/sbin/dnsmasq...  About an hour ago  Up About an hour ago         neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266
f6c508d26299  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           2 hours ago        Up 2 hours ago               neutron_ovs_agent
1a3a1498ea85  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           2 hours ago        Up 2 hours ago               neutron_metadata_agent
955e6c47dd80  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           kolla_start           2 hours ago        Up 2 hours ago               neutron_l3_agent
2699e7b00475  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         kolla_start           2 hours ago        Up 2 hours ago               neutron_dhcp
4d892b53d526  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           2 hours ago        Up 2 hours ago               neutron_api
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.21 "sudo podman ps | grep -i neutron"
922358f8cc13  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           /usr/sbin/keepali...  14 minutes ago     Up 14 minutes ago            neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad
eaedd62297a4  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         /usr/sbin/dnsmasq...  About an hour ago  Up About an hour ago         neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266
f6c508d26299  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           2 hours ago        Up 2 hours ago               neutron_ovs_agent
1a3a1498ea85  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           2 hours ago        Up 2 hours ago               neutron_metadata_agent
955e6c47dd80  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           kolla_start           2 hours ago        Up 2 hours ago               neutron_l3_agent
2699e7b00475  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         kolla_start           2 hours ago        Up 2 hours ago               neutron_dhcp
4d892b53d526  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           2 hours ago        Up 2 hours ago               neutron_api
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.14 "sudo podman ps | grep -i neutron"
^C
(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.15 "sudo podman ps | grep -i neutron"
edddc818df24  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           /usr/sbin/keepali...  14 minutes ago     Up 14 minutes ago            neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad
736f825342b3  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         /usr/sbin/dnsmasq...  About an hour ago  Up About an hour ago         neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266
3a55cce49e0b  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           2 hours ago        Up 2 hours ago               neutron_ovs_agent
55258961cb36  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           2 hours ago        Up 2 hours ago               neutron_metadata_agent
0ec4ddf9aa39  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           kolla_start           2 hours ago        Up 2 hours ago               neutron_l3_agent
00e3e24c5107  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         kolla_start           2 hours ago        Up 2 hours ago               neutron_dhcp
c469edaae51a  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           2 hours ago        Up 2 hours ago               neutron_api
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.11 "sudo podman ps | grep -i neutron"
ee70a6a377a5  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           /usr/sbin/keepali...  14 minutes ago     Up 14 minutes ago            neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad
b6715a07ab99  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         /usr/sbin/dnsmasq...  About an hour ago  Up About an hour ago         neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266
f33d4da542f3  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           2 hours ago        Up 2 hours ago               neutron_ovs_agent
9621b3fe1898  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           2 hours ago        Up 2 hours ago               neutron_metadata_agent
ea13d04586a9  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95           kolla_start           2 hours ago        Up 2 hours ago               neutron_l3_agent
f02c8d307e48  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94         kolla_start           2 hours ago        Up 2 hours ago               neutron_dhcp
11a933635eed  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           2 hours ago        Up 2 hours ago               neutron_api
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ date
Tue 28 Apr 06:02:36 EDT 2020
(overcloud) [stack@machine-16 ~]$
~~~

~~
(overcloud) [stack@machine-16 ~]$ cat /etc/rhosp-release 
Red Hat OpenStack Platform release 16.0.0 Beta (Train)
(overcloud) [stack@machine-16 ~]$ 
~~