Bug 1832720 - Stale neutron-haproxy-qdhcp containers after network delete in OVS deployed RHSOP-16
Summary: Stale neutron-haproxy-qdhcp containers after network delete in OVS deployed R...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 16.0 (Train)
Hardware: All
OS: All
high
medium
Target Milestone: rc
: 16.1 (Train on RHEL 8.2)
Assignee: Bernard Cafarelli
QA Contact: Alex Katz
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-07 07:49 UTC by Srinivas Atmakuri
Modified: 2023-12-15 17:54 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-heat-templates-11.3.2-0.20200603154815.1e9ce95.el8ost
Doc Type: Bug Fix
Doc Text:
Before this update, stale `neutron-haproxy-qdhcp-*` containers remained after you deleted the related network. With this update, all related containers are cleaned correctly when you delete a network.
Clone Of:
Environment:
Last Closed: 2020-07-29 07:52:21 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 730831 0 None MERGED Force container deletion if namespace does not exist in service_kill 2021-02-08 05:31:47 UTC
Red Hat Issue Tracker OSP-30815 0 None None None 2023-12-15 17:54:02 UTC
Red Hat Product Errata RHBA-2020:3148 0 None None None 2020-07-29 07:52:42 UTC

Description Srinivas Atmakuri 2020-05-07 07:49:37 UTC 
Description of problem:
Stale neutron-haproxy-qdhcp containers after network delete in OVS deployed RHSOP-16

Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform - 16 (RHOSP-16)

How reproducible:
Always

Steps to Reproduce:
1.Create a network and subnet
2. Observer neutron-haproxy-qdhcp containers are created on the controller nodes
3. Run openstack network delete command without running subnet delete.

OpenStack can delete the subnet along with network when network delete command, but this is causing stale neutron-haproxy-qdhcp containers after successful network delete.
The issue is not seen when, we run subnet delete and then network delete, observed only when we directly run network delete without subnet delete.

Actual results:
Stale neutron-haproxy-qdhcp containers after network delete

Expected results:
neutron-haproxy-qdhcp containers deleted along with network

Additional info:
(overcloud) [stack@machine-16 ~]$ openstack network list
+--------------------------------------+----------------------------------------------------+--------------------------------------+
| ID                                   | Name                                               | Subnets                              |
+--------------------------------------+----------------------------------------------------+--------------------------------------+
| 41397823-881c-4b15-b644-99e6c84fc7ac | testnetwork                                        | 2703be0e-42dd-4131-81fb-9bf33323bb05 |
| 628fd8d9-1fc9-4e9c-b85b-6ae7026d23c5 | testnetwork                                        | baa02584-c9e6-4273-af5f-2180978b71d6 |
| 75da1efd-af10-44c2-9131-ab06b2e63572 | HA network tenant 77d1bc3c0cd946ad8b2da7c505041152 | 2bd84d85-eba3-4898-86f7-2320bf8ffc62 |
+--------------------------------------+----------------------------------------------------+--------------------------------------+
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ openstack subnet list
+--------------------------------------+---------------------------------------------------+--------------------------------------+------------------+
| ID                                   | Name                                              | Network                              | Subnet           |
+--------------------------------------+---------------------------------------------------+--------------------------------------+------------------+
| 2703be0e-42dd-4131-81fb-9bf33323bb05 | testsubnet                                        | 41397823-881c-4b15-b644-99e6c84fc7ac | 192.168.122.0/24 |
| 2bd84d85-eba3-4898-86f7-2320bf8ffc62 | HA subnet tenant 77d1bc3c0cd946ad8b2da7c505041152 | 75da1efd-af10-44c2-9131-ab06b2e63572 | 169.254.192.0/18 |
| baa02584-c9e6-4273-af5f-2180978b71d6 | testsubnet                                        | 628fd8d9-1fc9-4e9c-b85b-6ae7026d23c5 | 1.2.3.0/24       |
+--------------------------------------+---------------------------------------------------+--------------------------------------+------------------+
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ openstack network delete 628fd8d9-1fc9-4e9c-b85b-6ae7026d23c5
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.18 "sudo podman ps | grep -i neutron"
d18cfd228361  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         /bin/bash -c HAPR...  3 minutes ago  Up 3 minutes ago         neutron-haproxy-qdhcp-628fd8d9-1fc9-4e9c-b85b-6ae7026d23c5
bfc91e5ebf80  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         /bin/bash -c HAPR...  3 minutes ago  Up 3 minutes ago         neutron-haproxy-qdhcp-41397823-881c-4b15-b644-99e6c84fc7ac
7bfc871ecd20  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         /usr/sbin/dnsmasq...  3 minutes ago  Up 3 minutes ago         neutron-dnsmasq-qdhcp-41397823-881c-4b15-b644-99e6c84fc7ac
58d6afb42291  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           /bin/bash -c HAPR...  3 days ago     Up 3 days ago            neutron-haproxy-qrouter-9e331647-bcb2-47cf-9934-d6e4cec9303f
f217d8d03236  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           /usr/sbin/keepali...  3 days ago     Up 3 days ago            neutron-keepalived-qrouter-9e331647-bcb2-47cf-9934-d6e4cec9303f
6d0c3a980951  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           6 days ago     Up 6 days ago            neutron_ovs_agent
49181e973a21  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           6 days ago     Up 4 minutes ago         neutron_metadata_agent
900ac0970586  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           kolla_start           6 days ago     Up 6 days ago            neutron_l3_agent
2c613ad11090  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         kolla_start           6 days ago     Up 4 minutes ago         neutron_dhcp
4f2c3f26fe2d  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           6 days ago     Up 6 days ago            neutron_api
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.21 "sudo podman ps | grep -i neutron"
76cc415e3b02  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         /bin/bash -c HAPR...  5 minutes ago  Up 5 minutes ago         neutron-haproxy-qdhcp-41397823-881c-4b15-b644-99e6c84fc7ac
8926764333be  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         /bin/bash -c HAPR...  5 minutes ago  Up 5 minutes ago         neutron-haproxy-qdhcp-628fd8d9-1fc9-4e9c-b85b-6ae7026d23c5
6f17b8b1f075  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         /usr/sbin/dnsmasq...  5 minutes ago  Up 5 minutes ago         neutron-dnsmasq-qdhcp-41397823-881c-4b15-b644-99e6c84fc7ac
a5a8073b9b9b  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           /usr/sbin/keepali...  3 days ago     Up 3 days ago            neutron-keepalived-qrouter-9e331647-bcb2-47cf-9934-d6e4cec9303f
a44fcd8171c9  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           6 days ago     Up 6 days ago            neutron_ovs_agent
c110f791c0ef  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           6 days ago     Up 5 minutes ago         neutron_metadata_agent
c7add71c710d  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           kolla_start           6 days ago     Up 6 days ago            neutron_l3_agent
828e6e83788c  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         kolla_start           6 days ago     Up 5 minutes ago         neutron_dhcp
0af0b71a155c  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           6 days ago     Up 6 days ago            neutron_api
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.10 "sudo podman ps | grep -i neutron"
0e9578826372  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         /usr/sbin/dnsmasq...  7 minutes ago   Up 7 minutes ago          neutron-dnsmasq-qdhcp-41397823-881c-4b15-b644-99e6c84fc7ac
4f25c1257454  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         /bin/bash -c HAPR...  13 minutes ago  Up 13 minutes ago         neutron-haproxy-qdhcp-41397823-881c-4b15-b644-99e6c84fc7ac
56140cfff61e  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         /bin/bash -c HAPR...  13 minutes ago  Up 13 minutes ago         neutron-haproxy-qdhcp-628fd8d9-1fc9-4e9c-b85b-6ae7026d23c5
c88ddfe1b9a3  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           /usr/sbin/keepali...  3 days ago      Up 3 days ago             neutron-keepalived-qrouter-9e331647-bcb2-47cf-9934-d6e4cec9303f
e731d5a1f6d2  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           6 days ago      Up 6 days ago             neutron_ovs_agent
e01ca90fbbfe  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           6 days ago      Up 8 minutes ago          neutron_metadata_agent
0717206bee8c  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           kolla_start           6 days ago      Up 6 days ago             neutron_l3_agent
457398e17505  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         kolla_start           6 days ago      Up 7 minutes ago          neutron_dhcp
2d44010c346c  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           6 days ago      Up 6 days ago             neutron_api
(overcloud) [stack@machine-16 ~]$ 

(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ openstack network list
+--------------------------------------+----------------------------------------------------+--------------------------------------+
| ID                                   | Name                                               | Subnets                              |
+--------------------------------------+----------------------------------------------------+--------------------------------------+
| 41397823-881c-4b15-b644-99e6c84fc7ac | testnetwork                                        | 2703be0e-42dd-4131-81fb-9bf33323bb05 |
| 75da1efd-af10-44c2-9131-ab06b2e63572 | HA network tenant 77d1bc3c0cd946ad8b2da7c505041152 | 2bd84d85-eba3-4898-86f7-2320bf8ffc62 |
+--------------------------------------+----------------------------------------------------+--------------------------------------+
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ openstack subnet delete 2703be0e-42dd-4131-81fb-9bf33323bb05
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ openstack network delete 41397823-881c-4b15-b644-99e6c84fc7ac
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.18 "sudo podman ps | grep -i neutron"
d18cfd228361  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         /bin/bash -c HAPR...  6 minutes ago  Up 6 minutes ago         neutron-haproxy-qdhcp-628fd8d9-1fc9-4e9c-b85b-6ae7026d23c5
f217d8d03236  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           /usr/sbin/keepali...  3 days ago     Up 3 days ago            neutron-keepalived-qrouter-9e331647-bcb2-47cf-9934-d6e4cec9303f
6d0c3a980951  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           6 days ago     Up 6 days ago            neutron_ovs_agent
49181e973a21  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           6 days ago     Up 6 minutes ago         neutron_metadata_agent
900ac0970586  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           kolla_start           6 days ago     Up 6 days ago            neutron_l3_agent
2c613ad11090  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         kolla_start           6 days ago     Up 6 minutes ago         neutron_dhcp
4f2c3f26fe2d  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           6 days ago     Up 6 days ago            neutron_api
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.21 "sudo podman ps | grep -i neutron"
13c518dd8925  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           /bin/bash -c HAPR...  51 seconds ago  Up 51 seconds ago         neutron-haproxy-qrouter-9e331647-bcb2-47cf-9934-d6e4cec9303f
8926764333be  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         /bin/bash -c HAPR...  7 minutes ago   Up 7 minutes ago          neutron-haproxy-qdhcp-628fd8d9-1fc9-4e9c-b85b-6ae7026d23c5
a5a8073b9b9b  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           /usr/sbin/keepali...  3 days ago      Up 3 days ago             neutron-keepalived-qrouter-9e331647-bcb2-47cf-9934-d6e4cec9303f
a44fcd8171c9  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           6 days ago      Up 6 days ago             neutron_ovs_agent
c110f791c0ef  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           6 days ago      Up 7 minutes ago          neutron_metadata_agent
c7add71c710d  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           kolla_start           6 days ago      Up 6 days ago             neutron_l3_agent
828e6e83788c  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         kolla_start           6 days ago      Up 7 minutes ago          neutron_dhcp
0af0b71a155c  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           6 days ago      Up 6 days ago             neutron_api
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ ssh heat-admin.24.10 "sudo podman ps | grep -i neutron"
56140cfff61e  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         /bin/bash -c HAPR...  15 minutes ago  Up 15 minutes ago         neutron-haproxy-qdhcp-628fd8d9-1fc9-4e9c-b85b-6ae7026d23c5
c88ddfe1b9a3  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           /usr/sbin/keepali...  3 days ago      Up 3 days ago             neutron-keepalived-qrouter-9e331647-bcb2-47cf-9934-d6e4cec9303f
e731d5a1f6d2  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.0-94  kolla_start           6 days ago      Up 6 days ago             neutron_ovs_agent
e01ca90fbbfe  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.0-95     kolla_start           6 days ago      Up 10 minutes ago         neutron_metadata_agent
0717206bee8c  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.0-98           kolla_start           6 days ago      Up 6 days ago             neutron_l3_agent
457398e17505  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.0-97         kolla_start           6 days ago      Up 9 minutes ago          neutron_dhcp
2d44010c346c  machine-16.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.0-94             kolla_start           6 days ago      Up 6 days ago             neutron_api
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ 
(overcloud) [stack@machine-16 ~]$ . stackrc 
(undercloud) [stack@machine-16 ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks               |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| a69bb9b9-f454-4ce5-821e-48397b2787c1 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.168.24.10 |
| 84c140fe-786c-443c-bc92-f8691d355fba | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.168.24.21 |
| a90ee7aa-fe08-459b-86c4-b6b554de3e69 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.168.24.18 |
| a7d74999-72ae-4583-abf8-c480e643a88c | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.16 |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
(undercloud) [stack@machine-16 ~]$ 
(undercloud) [stack@machine-16 ~]$ cat /etc/rhosp-release 
Red Hat OpenStack Platform release 16.0.0 Beta (Train)
(undercloud) [stack@machine-16 ~]$
Comment 1 Bernard Cafarelli 2020-05-07 13:34:44 UTC 
From quick check, this looks similar to bug #1816657 (for neutron-keepalived-qrouter sidecar containers)
Comment 2 Bernard Cafarelli 2020-05-26 14:33:13 UTC 
I cannot reproduce it on 16.0.2 (used to test bug #1816657):
(overcloud) [stack@undercloud-0 ~]$ openstack network create test && openstack subnet create testsub --network test --subnet-range 172.24.4.0/24 --gateway 172.24.4.1

(overcloud) [stack@undercloud-0 ~]$ openstack subnet list --network 
+--------------------------------------+---------+--------------------------------------+---------------+
| ID                                   | Name    | Network                              | Subnet        |
+--------------------------------------+---------+--------------------------------------+---------------+
| 19f5a241-245a-4935-bcf3-d1149b41ee18 | testsub | 35556fee-bb67-41f3-84b5-edca9a88d7b7 | 172.24.4.0/24 |
+--------------------------------------+---------+--------------------------------------+---------------+

On one controller:
[root@controller-0 neutron]# podman ps|grep qdhcp-35556fee-bb67-41f3-84b5-edca9a88d7b7
0c519ee9f606  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-dhcp-agent:20200416.1         /usr/sbin/dnsmasq...  5 minutes ago  Up 5 minutes ago         neutron-dnsmasq-qdhcp-35556fee-bb67-41f3-84b5-edca9a88d7b7


(overcloud) [stack@undercloud-0 ~]$ openstack network delete test
(overcloud) [stack@undercloud-0 ~]$ openstack subnet show 19f5a241-245a-4935-bcf3-d1149b41ee18
No Subnet found for 19f5a241-245a-4935-bcf3-d1149b41ee18

And on controller (same on the 2 other nodes):
[root@controller-0 neutron]# podman ps|grep qdhcp-35556fee-bb67-41f3-84b5-edca9a88d7b7
[root@controller-0 neutron]#


So subnet and dnsmasq sidecar containers are properly deleted. Also just deleting the subnet also properly exist the sidecar container.


Which specific 16.0 version is it? Also the kill-script log should be useful, it can be found in /var/log/containers/neutron/kill-script.log
Comment 5 Bernard Cafarelli 2020-05-28 09:23:46 UTC 
My bad, I just noticed I did not have haproxy sidecar containers in my test env. After enabling isolated metadata I can now see the issue and it is indeed different from keepalived sidecars from bug #1816657

(overcloud) [stack@undercloud-0 ~]$ openstack network create test && openstack subnet create testsub --network test --subnet-range 172.24.4.0/24 --gateway 172.24.4.1

On a controller:
[root@controller-2 neutron]# podman ps|grep 87f55e6c-506d-4bb1-806b-3ede5b737be4
effcf247d9b6  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-dhcp-agent:20200416.1         /bin/bash -c HAPR...  24 seconds ago  Up 24 seconds ago         neutron-haproxy-qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4
402df34a80d9  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-dhcp-agent:20200416.1         /usr/sbin/dnsmasq...  26 seconds ago  Up 26 seconds ago         neutron-dnsmasq-qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4


And kill-script.log when running "openstack network delete test":
+ SIG=9
+ PID=90994
++ ip netns identify 90994
+ NETNS=qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4
+ '[' xqdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4 == x ']'
+ CLI='nsenter --net=/run/netns/qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4 --preserve-credentials -m -t 1 podman'
+ '[' -f /proc/90994/cgroup ']'
++ awk 'BEGIN {FS="[-.]"} /name=/{print $3}' /proc/90994/cgroup
+ CT_ID=402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9
++ nsenter --net=/run/netns/qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4 --preserve-credentials -m -t 1 podman inspect -f '{{.Name}}' 402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9
+ CT_NAME=neutron-dnsmasq-qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4
+ case $SIG in
+ kill_container neutron-dnsmasq-qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4 402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9
+ add_date 'Stopping container neutron-dnsmasq-qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4 (402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9)'
++ date
+ echo 'Thu May 28 09:20:57 UTC 2020 Stopping container neutron-dnsmasq-qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4 (402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9)'
Thu May 28 09:20:57 UTC 2020 Stopping container neutron-dnsmasq-qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4 (402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9)
+ nsenter --net=/run/netns/qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4 --preserve-credentials -m -t 1 podman stop 402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9
402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9
+ add_date 'Deleting container neutron-dnsmasq-qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4 (402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9)'
++ date
+ echo 'Thu May 28 09:20:58 UTC 2020 Deleting container neutron-dnsmasq-qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4 (402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9)'
Thu May 28 09:20:58 UTC 2020 Deleting container neutron-dnsmasq-qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4 (402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9)
+ nsenter --net=/run/netns/qdhcp-87f55e6c-506d-4bb1-806b-3ede5b737be4 --preserve-credentials -m -t 1 podman rm 402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9
402df34a80d9eb142806ee2c999544c4cd8d02fb44aa6b377dbe61eba4910db9
+ exec
+ SIG=9
+ PID=91173
++ ip netns identify 91173
+ NETNS=
+ '[' x == x ']'
+ add_date 'No network namespace detected, exiting'
++ date
+ echo 'Thu May 28 09:21:02 UTC 2020 No network namespace detected, exiting'
Thu May 28 09:21:02 UTC 2020 No network namespace detected, exiting
+ exit 1
+ exec
+ SIG=9
+ PID=91173
++ ip netns identify 91173
+ NETNS=
+ '[' x == x ']'
+ add_date 'No network namespace detected, exiting'
++ date
+ echo 'Thu May 28 09:21:02 UTC 2020 No network namespace detected, exiting'
Thu May 28 09:21:02 UTC 2020 No network namespace detected, exiting
+ exit 1
+ exec

[root@controller-2 neutron]# ps 91173
    PID TTY      STAT   TIME COMMAND
  91173 ?        S      0:00 /usr/sbin/haproxy -Ws -f /var/lib/neutron/ns-metadata-proxy/87f55e6c-506d-4bb1-806b-3ede5b737be4.conf


So dnsmasq sidecar is stopped, namespace is deleted, but then it cannot be used anymore to delete the metadata proxy container
Comment 6 Bernard Cafarelli 2020-05-28 10:07:56 UTC 
I think this will be fixed by https://review.opendev.org/#/c/730831 testing locally (https://review.opendev.org/#/c/730744 being the parent fix for bug #1816657)
Comment 7 Bernard Cafarelli 2020-05-28 11:14:07 UTC 
With patch applied:
+ SIG=9
+ PID=632420
++ ip netns identify 632420
+ NETNS=
+ '[' x == x ']'
+ CLI='nsenter --all --preserve-credentials -t 1 podman'
+ SIG=9
+ '[' -f /proc/632420/cgroup ']'
++ awk 'BEGIN {FS="[-.]"} /name=/{print $3}' /proc/632420/cgroup
+ CT_ID=2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d
++ nsenter --all --preserve-credentials -t 1 podman inspect -f '{{.Name}}' 2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d
+ CT_NAME=neutron-haproxy-qdhcp-5452b2e9-d6f7-4ce0-8430-6b48c4fb8304
+ case $SIG in
+ kill_container neutron-haproxy-qdhcp-5452b2e9-d6f7-4ce0-8430-6b48c4fb8304 2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d
+ add_date 'Stopping container neutron-haproxy-qdhcp-5452b2e9-d6f7-4ce0-8430-6b48c4fb8304 (2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d)'
++ date
+ echo 'Thu May 28 11:11:38 UTC 2020 Stopping container neutron-haproxy-qdhcp-5452b2e9-d6f7-4ce0-8430-6b48c4fb8304 (2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d)'
Thu May 28 11:11:38 UTC 2020 Stopping container neutron-haproxy-qdhcp-5452b2e9-d6f7-4ce0-8430-6b48c4fb8304 (2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d)
+ nsenter --all --preserve-credentials -t 1 podman stop 2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d
2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d
+ add_date 'Deleting container neutron-haproxy-qdhcp-5452b2e9-d6f7-4ce0-8430-6b48c4fb8304 (2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d)'
++ date
+ echo 'Thu May 28 11:11:39 UTC 2020 Deleting container neutron-haproxy-qdhcp-5452b2e9-d6f7-4ce0-8430-6b48c4fb8304 (2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d)'
Thu May 28 11:11:39 UTC 2020 Deleting container neutron-haproxy-qdhcp-5452b2e9-d6f7-4ce0-8430-6b48c4fb8304 (2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d)
+ nsenter --all --preserve-credentials -t 1 podman rm 2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d
2f52e12e2f0b741e82da3ce15f0a978d04635bf55ea214c1772cbb402fe3173d
+ exec

And no remaining container linked to network ID remains on controller node
Comment 15 Alex McLeod 2020-06-16 12:32:22 UTC 
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.
Comment 17 errata-xmlrpc 2020-07-29 07:52:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3148
Note You need to log in before you can comment on or make changes to this bug.