Bug 1840611

Summary: [Kuryr] LB resources not deleted when are transitioned to ERROR
Product: OpenShift Container Platform Reporter: Maysa Macedo <mdemaced>
Component: NetworkingAssignee: Maysa Macedo <mdemaced>
Networking sub component: kuryr QA Contact: GenadiC <gcheresh>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: gcheresh, ltomasbo, rlobillo
Version: 4.5Keywords: UpcomingSprint
Target Milestone: ---   
Target Release: 4.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1840187 Environment:
Last Closed: 2020-06-23 00:57:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1840187    
Bug Blocks: 1846259    

Comment 3 rlobillo 2020-06-16 10:54:26 UTC 
Verified on 4.4.0-0.nightly-2020-06-14-142924 on OSP16 (RHOS_TRUNK-16.0-RHEL-8-20200513.n.1) with OVN.

1- Setup environment:

	$ oc new-project test
	$ oc run --image kuryr/demo demo
	$ oc run --image kuryr/demo demo-caller
	$ oc expose pod/demo-1-w87sm --port 80 --target-port 8080
	$ oc get all
	NAME                       READY   STATUS      RESTARTS   AGE
	pod/demo-1-deploy          0/1     Completed   0          66s
	pod/demo-1-w87sm           1/1     Running     0          46s
	pod/demo-caller-1-deploy   0/1     Completed   0          45s
	pod/demo-caller-1-twjlc    1/1     Running     0          25s

	NAME                                  DESIRED   CURRENT   READY   AGE
	replicationcontroller/demo-1          1         1         1       66s
	replicationcontroller/demo-caller-1   1         1         1       45s

	NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
	service/demo-1-w87sm   ClusterIP   172.30.17.214   <none>        80/TCP    4s

	NAME                                             REVISION   DESIRED   CURRENT   TRIGGERED BY
	deploymentconfig.apps.openshift.io/demo          1          1         1         config
	deploymentconfig.apps.openshift.io/demo-caller   1          1         1         config

	$ oc rsh pod/demo-caller-1-twjlc curl 172.30.17.214
	demo-1-w87sm: HELLO! I AM ALIVE!!!

2- Transition pool to ERROR state:

	$ openstack loadbalancer show test/demo-1-w87sm
	+---------------------+--------------------------------------+
	| Field               | Value                                |
	+---------------------+--------------------------------------+
	| admin_state_up      | True                                 |
	| created_at          | 2020-06-16T07:26:42                  |
	| description         |                                      |
	| flavor_id           | None                                 |
	| id                  | 678cb62c-6600-4c9e-873a-f5d0d13ee97e |
	| listeners           | ff596a87-7c2b-48ce-8cd3-8f2e24e7333f |
	| name                | test/demo-1-w87sm                    |
	| operating_status    | ONLINE                               |
	| pools               | 269b609f-2160-432f-9269-bf81753a5c5f |
	| project_id          | 1022ce3801a445df869b61b032d08925     |
	| provider            | ovn                                  |
	| provisioning_status | ACTIVE                               |
	| updated_at          | 2020-06-16T07:26:59                  |
	| vip_address         | 172.30.17.214                        |
	| vip_network_id      | b343be81-51ac-4e70-a293-a693c26605db |
	| vip_port_id         | 86dea4e5-2b1c-4139-b1a2-3e8bec1b78f6 |
	| vip_qos_policy_id   | None                                 |
	| vip_subnet_id       | 4018bc0c-8f2d-492e-bbe9-3f9de7682c3f |
	+---------------------+--------------------------------------+

	MariaDB [octavia]> update pool set provisioning_status='ERROR' where id='269b609f-2160-432f-9269-bf81753a5c5f';
	Query OK, 1 row affected (0.003 sec)
	Rows matched: 1  Changed: 1  Warnings: 0

	$ openstack loadbalancer pool show 269b609f-2160-432f-9269-bf81753a5c5f
	+----------------------+--------------------------------------+
	| Field                | Value                                |
	+----------------------+--------------------------------------+
	| admin_state_up       | True                                 |
	| created_at           | 2020-06-16T07:26:58                  |
	| description          |                                      |
	| healthmonitor_id     |                                      |
	| id                   | 269b609f-2160-432f-9269-bf81753a5c5f |
	| lb_algorithm         | SOURCE_IP_PORT                       |
	| listeners            | ff596a87-7c2b-48ce-8cd3-8f2e24e7333f |
	| loadbalancers        | 678cb62c-6600-4c9e-873a-f5d0d13ee97e |
	| members              | d8acb4f0-b0fa-4ae2-a91a-e6acd03b5868 |
	| name                 | test/demo-1-w87sm:TCP:80             |
	| operating_status     | ONLINE                               |
	| project_id           | 1022ce3801a445df869b61b032d08925     |
	| protocol             | TCP                                  |
	| provisioning_status  | ERROR                                |
	| session_persistence  | None                                 |
	| updated_at           | 2020-06-16T07:26:59                  |
	| tls_container_ref    | None                                 |
	| ca_tls_container_ref | None                                 |
	| crl_container_ref    | None                                 |
	| tls_enabled          | False                                |
	+----------------------+--------------------------------------+

	$ oc edit endpoints -n test #remove kuryr annotations to trigger kuryr-controller recalculation.
	endpoints/demo-1-w87sm edited

	# New pool assigned to the LB:
	$ openstack loadbalancer show test/demo-1-w87sm
	+---------------------+--------------------------------------+
	| Field               | Value                                |
	+---------------------+--------------------------------------+
	| admin_state_up      | True                                 |
	| created_at          | 2020-06-16T07:26:42                  |
	| description         |                                      |
	| flavor_id           | None                                 |
	| id                  | 678cb62c-6600-4c9e-873a-f5d0d13ee97e |
	| listeners           | ff596a87-7c2b-48ce-8cd3-8f2e24e7333f |
	| name                | test/demo-1-w87sm                    |
	| operating_status    | ONLINE                               |
	| pools               | b93f80fc-dc9c-4c10-b0e1-6a320ca6eba3 |
	| project_id          | 1022ce3801a445df869b61b032d08925     |
	| provider            | ovn                                  |
	| provisioning_status | ACTIVE                               |
	| updated_at          | 2020-06-16T07:38:34                  |
	| vip_address         | 172.30.17.214                        |
	| vip_network_id      | b343be81-51ac-4e70-a293-a693c26605db |
	| vip_port_id         | 86dea4e5-2b1c-4139-b1a2-3e8bec1b78f6 |
	| vip_qos_policy_id   | None                                 |
	| vip_subnet_id       | 4018bc0c-8f2d-492e-bbe9-3f9de7682c3f |
	+---------------------+--------------------------------------+
	$ oc rsh pod/demo-caller-1-twjlc curl 172.30.17.214
	demo-1-w87sm: HELLO! I AM ALIVE!!!

	# Old pool removed:
	$ openstack loadbalancer pool show 269b609f-2160-432f-9269-bf81753a5c5f
	Unable to locate 269b609f-2160-432f-9269-bf81753a5c5f in pools

	$ openstack loadbalancer pool show b93f80fc-dc9c-4c10-b0e1-6a320ca6eba3
	+----------------------+--------------------------------------+
	| Field                | Value                                |
	+----------------------+--------------------------------------+
	| admin_state_up       | True                                 |
	| created_at           | 2020-06-16T07:38:32                  |
	| description          |                                      |
	| healthmonitor_id     |                                      |
	| id                   | b93f80fc-dc9c-4c10-b0e1-6a320ca6eba3 |
	| lb_algorithm         | SOURCE_IP_PORT                       |
	| listeners            | ff596a87-7c2b-48ce-8cd3-8f2e24e7333f |
	| loadbalancers        | 678cb62c-6600-4c9e-873a-f5d0d13ee97e |
	| members              | 03cc141c-2cd4-4088-8681-2fbb11fb702b |
	| name                 | test/demo-1-w87sm:TCP:80             |
	| operating_status     | ONLINE                               |
	| project_id           | 1022ce3801a445df869b61b032d08925     |
	| protocol             | TCP                                  |
	| provisioning_status  | ACTIVE                               |
	| session_persistence  | None                                 |
	| updated_at           | 2020-06-16T07:38:34                  |
	| tls_container_ref    | None                                 |
	| ca_tls_container_ref | None                                 |
	| crl_container_ref    | None                                 |
	| tls_enabled          | False                                |
	+----------------------+--------------------------------------+

3- Transition member to ERROR state:

	$ openstack loadbalancer member list b93f80fc-dc9c-4c10-b0e1-6a320ca6eba3 -c id -f value
	03cc141c-2cd4-4088-8681-2fbb11fb702b

	MariaDB [octavia]> update member set provisioning_status='ERROR' where id='03cc141c-2cd4-4088-8681-2fbb11fb702b';
	Query OK, 1 row affected (0.003 sec)
	Rows matched: 1  Changed: 1  Warnings: 0

	$ openstack loadbalancer member show b93f80fc-dc9c-4c10-b0e1-6a320ca6eba3 03cc141c-2cd4-4088-8681-2fbb11fb702b
	+---------------------+--------------------------------------+
	| Field               | Value                                |
	+---------------------+--------------------------------------+
	| address             | 10.128.114.111                       |
	| admin_state_up      | True                                 |
	| created_at          | 2020-06-16T07:38:33                  |
	| id                  | 03cc141c-2cd4-4088-8681-2fbb11fb702b |
	| name                | test/demo-1-w87sm:8080               |
	| operating_status    | NO_MONITOR                           |
	| project_id          | 1022ce3801a445df869b61b032d08925     |
	| protocol_port       | 8080                                 |
	| provisioning_status | ERROR                                |
	| subnet_id           | f6d89651-5ecd-480c-b99a-51a5b59c497f |
	| updated_at          | 2020-06-16T07:38:34                  |
	| weight              | 1                                    |
	| monitor_port        | None                                 |
	| monitor_address     | None                                 |
	| backup              | False                                |
	+---------------------+--------------------------------------+

	$ oc edit endpoints -n test
	endpoints/demo-1-w87sm edited

	# New member created:
	$ openstack loadbalancer member list b93f80fc-dc9c-4c10-b0e1-6a320ca6eba3 -c id -f value
	cf613234-a055-4ed4-a37c-24092e9a6bae

	# Old member removed:
	$ openstack loadbalancer member show b93f80fc-dc9c-4c10-b0e1-6a320ca6eba3 03cc141c-2cd4-4088-8681-2fbb11fb702b
	Unable to locate 03cc141c-2cd4-4088-8681-2fbb11fb702b in members

	$ oc rsh pod/demo-caller-1-twjlc curl 172.30.17.214
	demo-1-w87sm: HELLO! I AM ALIVE!!!

3- Transition listener to ERROR state:

$ openstack loadbalancer listener list  | grep demo
| ff596a87-7c2b-48ce-8cd3-8f2e24e7333f | b93f80fc-dc9c-4c10-b0e1-6a320ca6eba3 | test/demo-1-w87sm:TCP:80                                                            | 1022ce3801a445df869b61b032d08925 | TCP      |            80 | True           |
$ openstack loadbalancer listener show ff596a87-7c2b-48ce-8cd3-8f2e24e7333f
+-----------------------------+--------------------------------------+
| Field                       | Value                                |
+-----------------------------+--------------------------------------+
| admin_state_up              | True                                 |
| connection_limit            | -1                                   |
| created_at                  | 2020-06-16T07:26:57                  |
| default_pool_id             | b93f80fc-dc9c-4c10-b0e1-6a320ca6eba3 |
| default_tls_container_ref   | None                                 |
| description                 |                                      |
| id                          | ff596a87-7c2b-48ce-8cd3-8f2e24e7333f |
| insert_headers              | None                                 |
| l7policies                  |                                      |
| loadbalancers               | 678cb62c-6600-4c9e-873a-f5d0d13ee97e |
| name                        | test/demo-1-w87sm:TCP:80             |
| operating_status            | ONLINE                               |
| project_id                  | 1022ce3801a445df869b61b032d08925     |
| protocol                    | TCP                                  |
| protocol_port               | 80                                   |
| provisioning_status         | ACTIVE                               |
| sni_container_refs          | []                                   |
| timeout_client_data         | 50000                                |
| timeout_member_connect      | 5000                                 |
| timeout_member_data         | 50000                                |
| timeout_tcp_inspect         | 0                                    |
| updated_at                  | 2020-06-16T07:49:54                  |
| client_ca_tls_container_ref | None                                 |
| client_authentication       | NONE                                 |
| client_crl_container_ref    | None                                 |
| allowed_cidrs               | None                                 |
+-----------------------------+--------------------------------------+

MariaDB [octavia]> update listener set provisioning_status='ERROR' where id='ff596a87-7c2b-48ce-8cd3-8f2e24e7333f';
Query OK, 1 row affected (0.002 sec)
Rows matched: 1  Changed: 1  Warnings: 0

$ openstack loadbalancer listener show ff596a87-7c2b-48ce-8cd3-8f2e24e7333f
+-----------------------------+--------------------------------------+
| Field                       | Value                                |
+-----------------------------+--------------------------------------+
| admin_state_up              | True                                 |
| connection_limit            | -1                                   |
| created_at                  | 2020-06-16T07:26:57                  |
| default_pool_id             | b93f80fc-dc9c-4c10-b0e1-6a320ca6eba3 |
| default_tls_container_ref   | None                                 |
| description                 |                                      |
| id                          | ff596a87-7c2b-48ce-8cd3-8f2e24e7333f |
| insert_headers              | None                                 |
| l7policies                  |                                      |
| loadbalancers               | 678cb62c-6600-4c9e-873a-f5d0d13ee97e |
| name                        | test/demo-1-w87sm:TCP:80             |
| operating_status            | ONLINE                               |
| project_id                  | 1022ce3801a445df869b61b032d08925     |
| protocol                    | TCP                                  |
| protocol_port               | 80                                   |
| provisioning_status         | ERROR                                |
| sni_container_refs          | []                                   |
| timeout_client_data         | 50000                                |
| timeout_member_connect      | 5000                                 |
| timeout_member_data         | 50000                                |
| timeout_tcp_inspect         | 0                                    |
| updated_at                  | 2020-06-16T07:49:54                  |
| client_ca_tls_container_ref | None                                 |
| client_authentication       | NONE                                 |
| client_crl_container_ref    | None                                 |
| allowed_cidrs               | None                                 |
+-----------------------------+--------------------------------------+

$ oc edit endpoints -n test
endpoints/demo-1-w87sm edited

$ openstack loadbalancer listener list  | grep demo
| bec7b622-b5dd-42ba-812a-6f6384cfff5a | 01335888-41ae-48f3-befc-abb6dff44dc3 | test/demo-1-w87sm:TCP:80                                                            | 1022ce3801a445df869b61b032d08925 | TCP      |            80 | True           |

# Old listener removed:
$ openstack loadbalancer listener show ff596a87-7c2b-48ce-8cd3-8f2e24e7333f
Unable to locate ff596a87-7c2b-48ce-8cd3-8f2e24e7333f in listeners

# New listener created:
$ openstack loadbalancer listener show bec7b622-b5dd-42ba-812a-6f6384cfff5a
+-----------------------------+--------------------------------------+
| Field                       | Value                                |
+-----------------------------+--------------------------------------+
| admin_state_up              | True                                 |
| connection_limit            | -1                                   |
| created_at                  | 2020-06-16T07:56:13                  |
| default_pool_id             | 01335888-41ae-48f3-befc-abb6dff44dc3 |
| default_tls_container_ref   | None                                 |
| description                 |                                      |
| id                          | bec7b622-b5dd-42ba-812a-6f6384cfff5a |
| insert_headers              | None                                 |
| l7policies                  |                                      |
| loadbalancers               | 678cb62c-6600-4c9e-873a-f5d0d13ee97e |
| name                        | test/demo-1-w87sm:TCP:80             |
| operating_status            | ONLINE                               |
| project_id                  | 1022ce3801a445df869b61b032d08925     |
| protocol                    | TCP                                  |
| protocol_port               | 80                                   |
| provisioning_status         | ACTIVE                               |
| sni_container_refs          | []                                   |
| timeout_client_data         | 50000                                |
| timeout_member_connect      | 5000                                 |
| timeout_member_data         | 50000                                |
| timeout_tcp_inspect         | 0                                    |
| updated_at                  | 2020-06-16T07:56:17                  |
| client_ca_tls_container_ref | None                                 |
| client_authentication       | NONE                                 |
| client_crl_container_ref    | None                                 |
| allowed_cidrs               | None                                 |
+-----------------------------+--------------------------------------+

$ oc rsh pod/demo-caller-1-twjlc curl 172.30.17.214
demo-1-w87sm: HELLO! I AM ALIVE!!!
Comment 5 errata-xmlrpc 2020-06-23 00:57:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2580