Description of problem: Stale neutron-keepalived-qrouter containers after the router delete in OVS deployed RHSOP-16 Version-Release number of selected component (if applicable): Red Hat Openstack Platform - 16 (RHOSP-16) How reproducible: Always Steps to Reproduce: 1. Create a router 2. Attach a gateway and detach 3. Now, delete the router Actual results: neutron-keepalived-qrouter-$ID present in UP state on the controllers nodes Expected results: neutron-keepalived-qrouter-$ID containers also deleted with router delete Additional info: (overcloud) [stack@machine-16 ~]$ openstack network list +--------------------------------------+----------------------------------------------------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------------------------------------------------+--------------------------------------+ | 09881516-cd27-45bb-8eeb-25de315fe8ee | HA network tenant 671470eaddec445db420e9cf5bddd969 | cee2c86e-2aab-46e0-940a-6e4aa60df233 | | 8a0de287-0c31-494d-b221-72752914b266 | testnetwork | 8826a9f1-4beb-4bef-bac4-bae29037090f | +--------------------------------------+----------------------------------------------------+--------------------------------------+ (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ openstack subnet list +--------------------------------------+---------------------------------------------------+--------------------------------------+------------------+ | ID | Name | Network | Subnet | +--------------------------------------+---------------------------------------------------+--------------------------------------+------------------+ | 8826a9f1-4beb-4bef-bac4-bae29037090f | testsubnet | 8a0de287-0c31-494d-b221-72752914b266 | 192.168.122.0/24 | | cee2c86e-2aab-46e0-940a-6e4aa60df233 | HA subnet tenant 671470eaddec445db420e9cf5bddd969 | 09881516-cd27-45bb-8eeb-25de315fe8ee | 169.254.192.0/18 | +--------------------------------------+---------------------------------------------------+--------------------------------------+------------------+ (overcloud) [stack@machine-16 ~]$ openstack router list +--------------------------------------+-------+--------+-------+----------------------------------+------+ | ID | Name | Status | State | Project | HA | +--------------------------------------+-------+--------+-------+----------------------------------+------+ | 253b1c97-4105-4bf6-9346-374b34fb92ad | route | ACTIVE | UP | 671470eaddec445db420e9cf5bddd969 | True | +--------------------------------------+-------+--------+-------+----------------------------------+------+ (overcloud) [stack@machine-16 ~]$ (undercloud) [stack@machine-16 ~]$ nova list +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+ | 6b6cce33-c2a8-43be-9cd5-fbce94c1add0 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.168.24.11 | | 9220f843-dd8c-41d5-a1c9-ad88d148ead7 | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.168.24.15 | | 7fad735e-076a-4e29-a6e6-251b7fea79ba | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.168.24.21 | | 55c88777-19bb-4402-9a09-b6599b6d328c | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.168.24.7 | +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+ (undercloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.11 "sudo podman ps | grep -i neutron" ee70a6a377a5 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 /usr/sbin/keepali... 7 minutes ago Up 7 minutes ago neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad b6715a07ab99 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 /usr/sbin/dnsmasq... About an hour ago Up About an hour ago neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266 f33d4da542f3 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94 kolla_start About an hour ago Up About an hour ago neutron_ovs_agent 9621b3fe1898 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95 kolla_start About an hour ago Up About an hour ago neutron_metadata_agent ea13d04586a9 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 kolla_start About an hour ago Up About an hour ago neutron_l3_agent f02c8d307e48 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 kolla_start About an hour ago Up About an hour ago neutron_dhcp 11a933635eed machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_api (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.15 "sudo podman ps | grep -i neutron" 515d7b57d109 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 /bin/bash -c HAPR... 7 minutes ago Up 7 minutes ago neutron-haproxy-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad edddc818df24 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 /usr/sbin/keepali... 7 minutes ago Up 7 minutes ago neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad 736f825342b3 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 /usr/sbin/dnsmasq... About an hour ago Up About an hour ago neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266 3a55cce49e0b machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94 kolla_start About an hour ago Up About an hour ago neutron_ovs_agent 55258961cb36 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95 kolla_start About an hour ago Up About an hour ago neutron_metadata_agent 0ec4ddf9aa39 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 kolla_start About an hour ago Up About an hour ago neutron_l3_agent 00e3e24c5107 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 kolla_start About an hour ago Up About an hour ago neutron_dhcp c469edaae51a machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_api (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.21 "sudo podman ps | grep -i neutron" 922358f8cc13 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 /usr/sbin/keepali... 7 minutes ago Up 7 minutes ago neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad eaedd62297a4 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 /usr/sbin/dnsmasq... About an hour ago Up About an hour ago neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266 f6c508d26299 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94 kolla_start About an hour ago Up About an hour ago neutron_ovs_agent 1a3a1498ea85 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95 kolla_start About an hour ago Up About an hour ago neutron_metadata_agent 955e6c47dd80 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 kolla_start About an hour ago Up About an hour ago neutron_l3_agent 2699e7b00475 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 kolla_start About an hour ago Up About an hour ago neutron_dhcp 4d892b53d526 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_api openstack router unset --external-gateway route (overcloud) [stack@machine-16 ~]$ openstack router delete route (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ openstack router list (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.21 "sudo podman ps | grep -i neutron" 922358f8cc13 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 /usr/sbin/keepali... 14 minutes ago Up 14 minutes ago neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad eaedd62297a4 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 /usr/sbin/dnsmasq... About an hour ago Up About an hour ago neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266 f6c508d26299 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_ovs_agent 1a3a1498ea85 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95 kolla_start 2 hours ago Up 2 hours ago neutron_metadata_agent 955e6c47dd80 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 kolla_start 2 hours ago Up 2 hours ago neutron_l3_agent 2699e7b00475 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_dhcp 4d892b53d526 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_api (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.21 "sudo podman ps | grep -i neutron" 922358f8cc13 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 /usr/sbin/keepali... 14 minutes ago Up 14 minutes ago neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad eaedd62297a4 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 /usr/sbin/dnsmasq... About an hour ago Up About an hour ago neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266 f6c508d26299 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_ovs_agent 1a3a1498ea85 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95 kolla_start 2 hours ago Up 2 hours ago neutron_metadata_agent 955e6c47dd80 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 kolla_start 2 hours ago Up 2 hours ago neutron_l3_agent 2699e7b00475 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_dhcp 4d892b53d526 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_api (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.14 "sudo podman ps | grep -i neutron" ^C (overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.15 "sudo podman ps | grep -i neutron" edddc818df24 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 /usr/sbin/keepali... 14 minutes ago Up 14 minutes ago neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad 736f825342b3 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 /usr/sbin/dnsmasq... About an hour ago Up About an hour ago neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266 3a55cce49e0b machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_ovs_agent 55258961cb36 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95 kolla_start 2 hours ago Up 2 hours ago neutron_metadata_agent 0ec4ddf9aa39 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 kolla_start 2 hours ago Up 2 hours ago neutron_l3_agent 00e3e24c5107 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_dhcp c469edaae51a machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_api (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.11 "sudo podman ps | grep -i neutron" ee70a6a377a5 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 /usr/sbin/keepali... 14 minutes ago Up 14 minutes ago neutron-keepalived-qrouter-253b1c97-4105-4bf6-9346-374b34fb92ad b6715a07ab99 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 /usr/sbin/dnsmasq... About an hour ago Up About an hour ago neutron-dnsmasq-qdhcp-8a0de287-0c31-494d-b221-72752914b266 f33d4da542f3 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_ovs_agent 9621b3fe1898 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95 kolla_start 2 hours ago Up 2 hours ago neutron_metadata_agent ea13d04586a9 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-95 kolla_start 2 hours ago Up 2 hours ago neutron_l3_agent f02c8d307e48 machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_dhcp 11a933635eed machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94 kolla_start 2 hours ago Up 2 hours ago neutron_api (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ (overcloud) [stack@machine-16 ~]$ date Tue 28 Apr 06:02:36 EDT 2020 (overcloud) [stack@machine-16 ~]$ ~~~ ~~ (overcloud) [stack@machine-16 ~]$ cat /etc/rhosp-release Red Hat OpenStack Platform release 16.0.0 Beta (Train) (overcloud) [stack@machine-16 ~]$ ~~