Bug 1938282 - [4.9] Kuryr won't remove LB members on Endpoints object removal
Summary: [4.9] Kuryr won't remove LB members on Endpoints object removal
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.8
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
: 4.9.0
Assignee: Robin Cernin
QA Contact: rlobillo
URL:
Whiteboard:
Depends On:
Blocks: 1995013
TreeView+ depends on / blocked
 
Reported: 2021-03-12 17:09 UTC by Michał Dulko
Modified: 2021-10-18 17:29 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-18 17:29:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 549 0 None None None 2021-08-17 22:50:15 UTC
Launchpad 1939512 0 None None None 2021-08-11 07:08:09 UTC
OpenStack gerrit 804205 0 None None None 2021-08-16 22:45:14 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:29:59 UTC

Description Michał Dulko 2021-03-12 17:09:59 UTC
Description of problem:
For selectorless services - i.e. the ones when it's user's responsibility to maintain the Endpoints object Kuryr is not reacting to the deletion of the Endpoints objects and not removing the LB members when it is gone.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create a Service without selectors.
2. Create some kuryr/demo pod.
3. Create an Endpoints object related to that Service and put IP of the pod from #2 there.
4. Check connectivity to the service. It should work as Kuryr should create an LB related to that Service and put pod from #2 as a member.
5. Delete the Endpoints object, wait ~1 minute.
6. Test connectivity to the Service.

Actual results:
The service is reachable.

Expected results:
The service should be not reachable, the LB should have no members or be deleted altogether.

Additional info:

Comment 10 rlobillo 2021-09-02 11:03:03 UTC
Verified on 4.9.0-0.nightly-2021-08-31-123131 on top of OSP16.1 (RHOS-16.1-RHEL-8-20210604.n.0) with OVN-Octavia.

# Given below project:

$ oc get all -n demo -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP               NODE                          NOMINATED NODE   READINESS GATES
pod/demo-7897db69cc-hrgzb   1/1     Running   0          20h   10.128.130.64    ostest-87ddk-worker-0-2bhm6   <none>           <none>
pod/demo-7897db69cc-l826k   1/1     Running   0          20h   10.128.130.57    ostest-87ddk-worker-0-2bhm6   <none>           <none>
pod/demo-7897db69cc-pp4jj   1/1     Running   0          20h   10.128.130.212   ostest-87ddk-worker-0-rsd89   <none>           <none>

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE   SELECTOR
service/my-service   ClusterIP   172.30.107.36   <none>        8080/TCP   81s   <none>

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES               SELECTOR
deployment.apps/demo   3/3     3            3           20h   demo         quay.io/kuryr/demo   app=demo

NAME                              DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES               SELECTOR
replicaset.apps/demo-7897db69cc   3         3         3       20h   demo         quay.io/kuryr/demo   app=demo,pod-template-hash=7897db69cc



# A service without selector is created:

$ cat service_wo_selectors.yaml 
apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: demo
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
$ oc apply -f service_wo_selectors.yaml 
service/my-service created

# No lb is created on openstack as there are no endpoints:

$ openstack loadbalancer show demo/my-service
Unable to locate demo/my-service in loadbalancers

# Endpoints with the specific PODS IPs is added to my-service service:

$ cat endpoints.yaml 
apiVersion: v1
kind: Endpoints
metadata:
  name: my-service
  namespace: demo
subsets:
  - addresses:
      - ip: 10.128.130.64
      - ip: 10.128.130.57
      - ip: 10.128.130.212
    ports:
      - port: 8080
(shiftstack) [stack@undercloud-0 BZ1938282]$ oc apply -f endpoints.yaml 
endpoints/my-service created

# The lb is created on openstack with below endpoints:

$ openstack loadbalancer show demo/my-service
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2021-09-02T10:54:15                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 54b3ec22-66a3-4eef-a7d1-d221ae576512 |
| listeners           | d22fcb0f-5eaf-41e0-9257-39ca8cee4275 |
| name                | demo/my-service                      |
| operating_status    | ONLINE                               |
| pools               | 3d512969-2b26-4dd9-9f30-17cbbb645033 |
| project_id          | 8493e10827254256b31b4b5ecefc8420     |
| provider            | ovn                                  |
| provisioning_status | ACTIVE                               |
| updated_at          | 2021-09-02T10:54:49                  |
| vip_address         | 172.30.107.36                        |
| vip_network_id      | 755e0bef-ad65-46ac-abac-3e183daf050e |
| vip_port_id         | 16824d3b-e9ae-4e73-95c0-e712502c9079 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | acc4583a-e935-4a2b-a06e-db1f3f4a77f5 |
+---------------------+--------------------------------------+

$ openstack loadbalancer member list $(openstack loadbalancer show demo/my-service -c pools -f value)
+--------------------------------------+----------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+
| id                                   | name                 | project_id                       | provisioning_status | address        | protocol_port | operating_status | weight |
+--------------------------------------+----------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+
| 0e73175a-5794-489e-ac34-3544e565b8c5 | demo/my-service:8080 | 8493e10827254256b31b4b5ecefc8420 | ACTIVE              | 10.128.130.64  |          8080 | NO_MONITOR       |      1 |
| d0078a7d-0f22-4814-aa96-7b91d5c9e0dc | demo/my-service:8080 | 8493e10827254256b31b4b5ecefc8420 | ACTIVE              | 10.128.130.57  |          8080 | NO_MONITOR       |      1 |
| 60ae48b2-b608-4241-96c7-4fffe503f455 | demo/my-service:8080 | 8493e10827254256b31b4b5ecefc8420 | ACTIVE              | 10.128.130.212 |          8080 | NO_MONITOR       |      1 |
+--------------------------------------+----------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+

# The svc is working as expected:

$ oc rsh demo-7897db69cc-hrgzb curl 172.30.107.36:8080
demo-7897db69cc-hrgzb: HELLO! I AM ALIVE!!!
$ oc rsh demo-7897db69cc-hrgzb curl 172.30.107.36:8080
demo-7897db69cc-l826k: HELLO! I AM ALIVE!!!


# Removing the enpoints object on OCP removes the members but not the loadbalancer.

$ oc delete endpoints my-service 
endpoints "my-service" deleted
$ openstack loadbalancer member list $(openstack loadbalancer show demo/my-service -c pools -f value)

$ openstack loadbalancer show demo/my-service                                                                                                     
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2021-09-02T10:54:15                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 54b3ec22-66a3-4eef-a7d1-d221ae576512 |
| listeners           | d22fcb0f-5eaf-41e0-9257-39ca8cee4275 |
| name                | demo/my-service                      |
| operating_status    | ONLINE                               |
| pools               | 3d512969-2b26-4dd9-9f30-17cbbb645033 |
| project_id          | 8493e10827254256b31b4b5ecefc8420     |
| provider            | ovn                                  |
| provisioning_status | ACTIVE                               |
| updated_at          | 2021-09-02T10:58:17                  |
| vip_address         | 172.30.107.36                        |
| vip_network_id      | 755e0bef-ad65-46ac-abac-3e183daf050e |
| vip_port_id         | 16824d3b-e9ae-4e73-95c0-e712502c9079 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | acc4583a-e935-4a2b-a06e-db1f3f4a77f5 |
+---------------------+--------------------------------------+


# Removing the service removes the loadbalancer:

$ oc delete service/my-service
service "my-service" deleted
$ openstack loadbalancer show demo/my-service
Unable to locate demo/my-service in loadbalancers

Comment 13 errata-xmlrpc 2021-10-18 17:29:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.