Bug 1846259 - [Kuryr] LB resources not deleted when are transitioned to ERROR
Summary: [Kuryr] LB resources not deleted when are transitioned to ERROR
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.3.z
Assignee: Maysa Macedo
QA Contact: GenadiC
URL:
Whiteboard:
Depends On: 1840611
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-11 08:41 UTC by OpenShift BugZilla Robot
Modified: 2020-07-01 15:02 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-01 15:02:34 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 276 None closed [release-4.3] Bug 1846259: Ensure LB resources with ERROR status are deleted 2020-08-20 08:44:27 UTC
Red Hat Product Errata RHBA-2020:2628 None None None 2020-07-01 15:02:49 UTC

Comment 3 rlobillo 2020-06-22 15:14:46 UTC
Verified on OCP4.3.0-0.nightly-2020-06-20-130223 on OSP16 (RHOS_TRUNK-16.0-RHEL-8-20200513.n.1) with OVN.

############################################################################
# 1. Setup environment.

oc new-project test && oc run --image kuryr/demo demo && oc run --image kuryr/demo demo-caller
oc expose pod/demo-1-dm552 --port 80 --target-port 8080


[stack@undercloud-0 ~]$ oc get all
NAME                       READY   STATUS      RESTARTS   AGE
pod/demo-1-deploy          0/1     Completed   0          4m25s
pod/demo-1-dm552           1/1     Running     0          3m59s
pod/demo-caller-1-deploy   0/1     Completed   0          4m25s
pod/demo-caller-1-pwl8k    1/1     Running     0          3m53s

NAME                                  DESIRED   CURRENT   READY   AGE
replicationcontroller/demo-1          1         1         1       4m25s
replicationcontroller/demo-caller-1   1         1         1       4m25s

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/demo-1-dm552   ClusterIP   172.30.241.93   <none>        80/TCP    114s

NAME                                             REVISION   DESIRED   CURRENT   TRIGGERED BY
deploymentconfig.apps.openshift.io/demo          1          1         1         config
deploymentconfig.apps.openshift.io/demo-caller   1          1         1         config
[stack@undercloud-0 ~]$ oc rsh pod/demo-caller-1-pwl8k curl 172.30.241.93
demo-1-dm552: HELLO! I AM ALIVE!!!

$ openstack loadbalancer show test/demo-1-dm552
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2020-06-22T14:23:29                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 42a27dcf-98a5-4241-b62f-268e16b55399 |
| listeners           | fe865c2b-eb72-45a8-8017-076a29ac4856 |
| name                | test/demo-1-dm552                    |
| operating_status    | ONLINE                               |
| pools               | 6954c127-b38f-42c6-9784-a94d07b3a100 |
| project_id          | cfeb3ef1735a45a1b5fc21f0d59a1fe6     |
| provider            | amphora                              |
| provisioning_status | ACTIVE                               |
| updated_at          | 2020-06-22T14:25:11                  |
| vip_address         | 172.30.241.93                        |
| vip_network_id      | 6bead3ae-b1cd-45be-adef-4f5b711aed45 |
| vip_port_id         | f1e52886-2236-40e2-96e0-821775a8132a |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | f41c1056-4342-48ee-b31d-a5319e31caf5 |
+---------------------+--------------------------------------+


############################################################################
# 2. Transition pool resource to ERROR and trigger recreation:

MariaDB [octavia]> update pool set provisioning_status='ERROR' where id='6954c127-b38f-42c6-9784-a94d07b3a100';
Query OK, 1 row affected (0.003 sec)
Rows matched: 1  Changed: 1  Warnings: 0


$ openstack loadbalancer pool show 6954c127-b38f-42c6-9784-a94d07b3a100
+----------------------+--------------------------------------+
| Field                | Value                                |
+----------------------+--------------------------------------+
| admin_state_up       | True                                 |
| created_at           | 2020-06-22T14:25:01                  |
| description          |                                      |
| healthmonitor_id     |                                      |
| id                   | 6954c127-b38f-42c6-9784-a94d07b3a100 |
| lb_algorithm         | ROUND_ROBIN                          |
| listeners            | fe865c2b-eb72-45a8-8017-076a29ac4856 |
| loadbalancers        | 42a27dcf-98a5-4241-b62f-268e16b55399 |
| members              | 243d263c-5e8d-4ea3-9281-114df2460737 |
| name                 | test/demo-1-dm552:TCP:80             |
| operating_status     | ONLINE                               |
| project_id           | cfeb3ef1735a45a1b5fc21f0d59a1fe6     |
| protocol             | TCP                                  |
| provisioning_status  | ERROR                                |
| session_persistence  | None                                 |
| updated_at           | 2020-06-22T14:25:08                  |
| tls_container_ref    | None                                 |
| ca_tls_container_ref | None                                 |
| crl_container_ref    | None                                 |
| tls_enabled          | False                                |
+----------------------+--------------------------------------+

$ oc edit endpoints # remove kuryr-controller annotations.


# ERROR pool is deleted and new one is created:
$ openstack loadbalancer pool show 6954c127-b38f-42c6-9784-a94d07b3a100
Unable to locate 6954c127-b38f-42c6-9784-a94d07b3a100 in pools

$ openstack loadbalancer show test/demo-1-dm552
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2020-06-22T14:23:29                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 42a27dcf-98a5-4241-b62f-268e16b55399 |
| listeners           | fe865c2b-eb72-45a8-8017-076a29ac4856 |
| name                | test/demo-1-dm552                    |
| operating_status    | ONLINE                               |
| pools               | 6ca15b4a-88a9-4b10-a902-b25166d3af21 |
| project_id          | cfeb3ef1735a45a1b5fc21f0d59a1fe6     |
| provider            | amphora                              |
| provisioning_status | ACTIVE                               |
| updated_at          | 2020-06-22T14:49:29                  |
| vip_address         | 172.30.241.93                        |
| vip_network_id      | 6bead3ae-b1cd-45be-adef-4f5b711aed45 |
| vip_port_id         | f1e52886-2236-40e2-96e0-821775a8132a |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | f41c1056-4342-48ee-b31d-a5319e31caf5 |
+---------------------+--------------------------------------+
$ openstack loadbalancer pool show 6ca15b4a-88a9-4b10-a902-b25166d3af21
+----------------------+--------------------------------------+
| Field                | Value                                |
+----------------------+--------------------------------------+
| admin_state_up       | True                                 |
| created_at           | 2020-06-22T14:49:21                  |
| description          |                                      |
| healthmonitor_id     |                                      |
| id                   | 6ca15b4a-88a9-4b10-a902-b25166d3af21 |
| lb_algorithm         | ROUND_ROBIN                          |
| listeners            | fe865c2b-eb72-45a8-8017-076a29ac4856 |
| loadbalancers        | 42a27dcf-98a5-4241-b62f-268e16b55399 |
| members              | c3056aca-b620-43d1-842d-cb7075d169a8 |
| name                 | test/demo-1-dm552:TCP:80             |
| operating_status     | ONLINE                               |
| project_id           | cfeb3ef1735a45a1b5fc21f0d59a1fe6     |
| protocol             | TCP                                  |
| provisioning_status  | ACTIVE                               |
| session_persistence  | None                                 |
| updated_at           | 2020-06-22T14:49:28                  |
| tls_container_ref    | None                                 |
| ca_tls_container_ref | None                                 |
| crl_container_ref    | None                                 |
| tls_enabled          | False                                |
+----------------------+--------------------------------------+

$ oc rsh pod/demo-caller-1-pwl8k curl 172.30.241.93
demo-1-dm552: HELLO! I AM ALIVE!!!


############################################################################
# 3 - Transition member resource to ERROR and trigger recreation:

MariaDB [octavia]> update member set provisioning_status='ERROR' where id='c3056aca-b620-43d1-842d-cb7075d169a8';
Query OK, 1 row affected (0.002 sec)
Rows matched: 1  Changed: 1  Warnings: 0

$ openstack loadbalancer member list 6ca15b4a-88a9-4b10-a902-b25166d3af21
+--------------------------------------+------------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+
| id                                   | name                   | project_id                       | provisioning_status | address        | protocol_port | operating_status | weight |
+--------------------------------------+------------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+
| c3056aca-b620-43d1-842d-cb7075d169a8 | test/demo-1-dm552:8080 | cfeb3ef1735a45a1b5fc21f0d59a1fe6 | ERROR               | 10.128.107.137 |          8080 | NO_MONITOR       |      1 |
+--------------------------------------+------------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+

$ oc edit endpoints # remove kuryr-controller annotations.

# ERROR member deleted and new one created:

$ openstack loadbalancer member list 6ca15b4a-88a9-4b10-a902-b25166d3af21
+--------------------------------------+------------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+
| id                                   | name                   | project_id                       | provisioning_status | address        | protocol_port | operating_status | weight |
+--------------------------------------+------------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+
| 49059175-5283-45fa-bef2-0f2836272ee8 | test/demo-1-dm552:8080 | cfeb3ef1735a45a1b5fc21f0d59a1fe6 | ACTIVE              | 10.128.107.137 |          8080 | NO_MONITOR       |      1 |
+--------------------------------------+------------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+

$ oc rsh pod/demo-caller-1-pwl8k curl 172.30.241.93
demo-1-dm552: HELLO! I AM ALIVE!!!

######################################
# 4 - Transition listener resource to ERROR and retrigger recreation:

$ openstack loadbalancer pool show 6ca15b4a-88a9-4b10-a902-b25166d3af21
+----------------------+--------------------------------------+
| Field                | Value                                |
+----------------------+--------------------------------------+
| admin_state_up       | True                                 |
| created_at           | 2020-06-22T14:49:21                  |
| description          |                                      |
| healthmonitor_id     |                                      |
| id                   | 6ca15b4a-88a9-4b10-a902-b25166d3af21 |
| lb_algorithm         | ROUND_ROBIN                          |
| listeners            | fe865c2b-eb72-45a8-8017-076a29ac4856 |
| loadbalancers        | 42a27dcf-98a5-4241-b62f-268e16b55399 |
| members              | 49059175-5283-45fa-bef2-0f2836272ee8 |
| name                 | test/demo-1-dm552:TCP:80             |
| operating_status     | ONLINE                               |
| project_id           | cfeb3ef1735a45a1b5fc21f0d59a1fe6     |
| protocol             | TCP                                  |
| provisioning_status  | ACTIVE                               |
| session_persistence  | None                                 |
| updated_at           | 2020-06-22T14:58:42                  |
| tls_container_ref    | None                                 |
| ca_tls_container_ref | None                                 |
| crl_container_ref    | None                                 |
| tls_enabled          | False                                |
+----------------------+--------------------------------------+

$ openstack loadbalancer listener show fe865c2b-eb72-45a8-8017-076a29ac4856
+-----------------------------+--------------------------------------+
| Field                       | Value                                |
+-----------------------------+--------------------------------------+
| admin_state_up              | True                                 |
| connection_limit            | -1                                   |
| created_at                  | 2020-06-22T14:24:56                  |
| default_pool_id             | 6ca15b4a-88a9-4b10-a902-b25166d3af21 |
| default_tls_container_ref   | None                                 |
| description                 |                                      |
| id                          | fe865c2b-eb72-45a8-8017-076a29ac4856 |
| insert_headers              | None                                 |
| l7policies                  |                                      |
| loadbalancers               | 42a27dcf-98a5-4241-b62f-268e16b55399 |
| name                        | test/demo-1-dm552:TCP:80             |
| operating_status            | ONLINE                               |
| project_id                  | cfeb3ef1735a45a1b5fc21f0d59a1fe6     |
| protocol                    | TCP                                  |
| protocol_port               | 80                                   |
| provisioning_status         | ACTIVE                               |
| sni_container_refs          | []                                   |
| timeout_client_data         | 50000                                |
| timeout_member_connect      | 5000                                 |
| timeout_member_data         | 50000                                |
| timeout_tcp_inspect         | 0                                    |
| updated_at                  | 2020-06-22T14:58:47                  |
| client_ca_tls_container_ref | None                                 |
| client_authentication       | NONE                                 |
| client_crl_container_ref    | None                                 |
| allowed_cidrs               | None                                 |
+-----------------------------+--------------------------------------+

MariaDB [octavia]> update listener set provisioning_status='ERROR' where id='fe865c2b-eb72-45a8-8017-076a29ac4856';
Query OK, 1 row affected (0.002 sec)
Rows matched: 1  Changed: 1  Warnings: 0

$ openstack loadbalancer listener show fe865c2b-eb72-45a8-8017-076a29ac4856
+-----------------------------+--------------------------------------+
| Field                       | Value                                |
+-----------------------------+--------------------------------------+
| admin_state_up              | True                                 |
| connection_limit            | -1                                   |
| created_at                  | 2020-06-22T14:24:56                  |
| default_pool_id             | 6ca15b4a-88a9-4b10-a902-b25166d3af21 |
| default_tls_container_ref   | None                                 |
| description                 |                                      |
| id                          | fe865c2b-eb72-45a8-8017-076a29ac4856 |
| insert_headers              | None                                 |
| l7policies                  |                                      |
| loadbalancers               | 42a27dcf-98a5-4241-b62f-268e16b55399 |
| name                        | test/demo-1-dm552:TCP:80             |
| operating_status            | ONLINE                               |
| project_id                  | cfeb3ef1735a45a1b5fc21f0d59a1fe6     |
| protocol                    | TCP                                  |
| protocol_port               | 80                                   |
| provisioning_status         | ERROR                                |
| sni_container_refs          | []                                   |
| timeout_client_data         | 50000                                |
| timeout_member_connect      | 5000                                 |
| timeout_member_data         | 50000                                |
| timeout_tcp_inspect         | 0                                    |
| updated_at                  | 2020-06-22T14:58:47                  |
| client_ca_tls_container_ref | None                                 |
| client_authentication       | NONE                                 |
| client_crl_container_ref    | None                                 |
| allowed_cidrs               | None                                 |
+-----------------------------+--------------------------------------+

$ oc edit endpoints # remove kuryr-controller annotations.
endpoints/demo-1-dm552 edited

# ERROR listener deleted and new one created:

$ openstack loadbalancer listener show fe865c2b-eb72-45a8-8017-076a29ac4856
Unable to locate fe865c2b-eb72-45a8-8017-076a29ac4856 in listeners
$ openstack loadbalancer listener show test/demo-1-dm552:TCP:80
+-----------------------------+--------------------------------------+
| Field                       | Value                                |
+-----------------------------+--------------------------------------+
| admin_state_up              | True                                 |
| connection_limit            | -1                                   |
| created_at                  | 2020-06-22T15:06:29                  |
| default_pool_id             | e47d21d0-cee3-4e44-aa9e-677829b6f7dc |
| default_tls_container_ref   | None                                 |
| description                 |                                      |
| id                          | 178c873e-9063-403d-b97b-529f665b84dc |
| insert_headers              | None                                 |
| l7policies                  |                                      |
| loadbalancers               | 42a27dcf-98a5-4241-b62f-268e16b55399 |
| name                        | test/demo-1-dm552:TCP:80             |
| operating_status            | ONLINE                               |
| project_id                  | cfeb3ef1735a45a1b5fc21f0d59a1fe6     |
| protocol                    | TCP                                  |
| protocol_port               | 80                                   |
| provisioning_status         | ACTIVE                               |
| sni_container_refs          | []                                   |
| timeout_client_data         | 50000                                |
| timeout_member_connect      | 5000                                 |
| timeout_member_data         | 50000                                |
| timeout_tcp_inspect         | 0                                    |
| updated_at                  | 2020-06-22T15:06:42                  |
| client_ca_tls_container_ref | None                                 |
| client_authentication       | NONE                                 |
| client_crl_container_ref    | None                                 |
| allowed_cidrs               | None                                 |
+-----------------------------+--------------------------------------+

$ oc rsh pod/demo-caller-1-pwl8k curl 172.30.241.93
demo-1-dm552: HELLO! I AM ALIVE!!!

kuryr-controller remained stable during the above tests.

Comment 5 errata-xmlrpc 2020-07-01 15:02:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2628


Note You need to log in before you can comment on or make changes to this bug.