Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1938282

Summary: [4.9] Kuryr won't remove LB members on Endpoints object removal
Product: OpenShift Container Platform Reporter: Michał Dulko <mdulko>
Component: NetworkingAssignee: Robin Cernin <rcernin>
Networking sub component: kuryr QA Contact: rlobillo
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: low CC: mdemaced, rcernin
Version: 4.8Keywords: Triaged
Target Milestone: ---   
Target Release: 4.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-18 17:29:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1995013    

Description Michał Dulko 2021-03-12 17:09:59 UTC 
Description of problem:
For selectorless services - i.e. the ones when it's user's responsibility to maintain the Endpoints object Kuryr is not reacting to the deletion of the Endpoints objects and not removing the LB members when it is gone.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create a Service without selectors.
2. Create some kuryr/demo pod.
3. Create an Endpoints object related to that Service and put IP of the pod from #2 there.
4. Check connectivity to the service. It should work as Kuryr should create an LB related to that Service and put pod from #2 as a member.
5. Delete the Endpoints object, wait ~1 minute.
6. Test connectivity to the Service.

Actual results:
The service is reachable.

Expected results:
The service should be not reachable, the LB should have no members or be deleted altogether.

Additional info:
Comment 10 rlobillo 2021-09-02 11:03:03 UTC 
Verified on 4.9.0-0.nightly-2021-08-31-123131 on top of OSP16.1 (RHOS-16.1-RHEL-8-20210604.n.0) with OVN-Octavia.

# Given below project:

$ oc get all -n demo -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP               NODE                          NOMINATED NODE   READINESS GATES
pod/demo-7897db69cc-hrgzb   1/1     Running   0          20h   10.128.130.64    ostest-87ddk-worker-0-2bhm6   <none>           <none>
pod/demo-7897db69cc-l826k   1/1     Running   0          20h   10.128.130.57    ostest-87ddk-worker-0-2bhm6   <none>           <none>
pod/demo-7897db69cc-pp4jj   1/1     Running   0          20h   10.128.130.212   ostest-87ddk-worker-0-rsd89   <none>           <none>

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE   SELECTOR
service/my-service   ClusterIP   172.30.107.36   <none>        8080/TCP   81s   <none>

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES               SELECTOR
deployment.apps/demo   3/3     3            3           20h   demo         quay.io/kuryr/demo   app=demo

NAME                              DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES               SELECTOR
replicaset.apps/demo-7897db69cc   3         3         3       20h   demo         quay.io/kuryr/demo   app=demo,pod-template-hash=7897db69cc



# A service without selector is created:

$ cat service_wo_selectors.yaml 
apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: demo
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
$ oc apply -f service_wo_selectors.yaml 
service/my-service created

# No lb is created on openstack as there are no endpoints:

$ openstack loadbalancer show demo/my-service
Unable to locate demo/my-service in loadbalancers

# Endpoints with the specific PODS IPs is added to my-service service:

$ cat endpoints.yaml 
apiVersion: v1
kind: Endpoints
metadata:
  name: my-service
  namespace: demo
subsets:
  - addresses:
      - ip: 10.128.130.64
      - ip: 10.128.130.57
      - ip: 10.128.130.212
    ports:
      - port: 8080
(shiftstack) [stack@undercloud-0 BZ1938282]$ oc apply -f endpoints.yaml 
endpoints/my-service created

# The lb is created on openstack with below endpoints:

$ openstack loadbalancer show demo/my-service
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2021-09-02T10:54:15                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 54b3ec22-66a3-4eef-a7d1-d221ae576512 |
| listeners           | d22fcb0f-5eaf-41e0-9257-39ca8cee4275 |
| name                | demo/my-service                      |
| operating_status    | ONLINE                               |
| pools               | 3d512969-2b26-4dd9-9f30-17cbbb645033 |
| project_id          | 8493e10827254256b31b4b5ecefc8420     |
| provider            | ovn                                  |
| provisioning_status | ACTIVE                               |
| updated_at          | 2021-09-02T10:54:49                  |
| vip_address         | 172.30.107.36                        |
| vip_network_id      | 755e0bef-ad65-46ac-abac-3e183daf050e |
| vip_port_id         | 16824d3b-e9ae-4e73-95c0-e712502c9079 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | acc4583a-e935-4a2b-a06e-db1f3f4a77f5 |
+---------------------+--------------------------------------+

$ openstack loadbalancer member list $(openstack loadbalancer show demo/my-service -c pools -f value)
+--------------------------------------+----------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+
| id                                   | name                 | project_id                       | provisioning_status | address        | protocol_port | operating_status | weight |
+--------------------------------------+----------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+
| 0e73175a-5794-489e-ac34-3544e565b8c5 | demo/my-service:8080 | 8493e10827254256b31b4b5ecefc8420 | ACTIVE              | 10.128.130.64  |          8080 | NO_MONITOR       |      1 |
| d0078a7d-0f22-4814-aa96-7b91d5c9e0dc | demo/my-service:8080 | 8493e10827254256b31b4b5ecefc8420 | ACTIVE              | 10.128.130.57  |          8080 | NO_MONITOR       |      1 |
| 60ae48b2-b608-4241-96c7-4fffe503f455 | demo/my-service:8080 | 8493e10827254256b31b4b5ecefc8420 | ACTIVE              | 10.128.130.212 |          8080 | NO_MONITOR       |      1 |
+--------------------------------------+----------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+

# The svc is working as expected:

$ oc rsh demo-7897db69cc-hrgzb curl 172.30.107.36:8080
demo-7897db69cc-hrgzb: HELLO! I AM ALIVE!!!
$ oc rsh demo-7897db69cc-hrgzb curl 172.30.107.36:8080
demo-7897db69cc-l826k: HELLO! I AM ALIVE!!!


# Removing the enpoints object on OCP removes the members but not the loadbalancer.

$ oc delete endpoints my-service 
endpoints "my-service" deleted
$ openstack loadbalancer member list $(openstack loadbalancer show demo/my-service -c pools -f value)

$ openstack loadbalancer show demo/my-service                                                                                                     
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2021-09-02T10:54:15                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 54b3ec22-66a3-4eef-a7d1-d221ae576512 |
| listeners           | d22fcb0f-5eaf-41e0-9257-39ca8cee4275 |
| name                | demo/my-service                      |
| operating_status    | ONLINE                               |
| pools               | 3d512969-2b26-4dd9-9f30-17cbbb645033 |
| project_id          | 8493e10827254256b31b4b5ecefc8420     |
| provider            | ovn                                  |
| provisioning_status | ACTIVE                               |
| updated_at          | 2021-09-02T10:58:17                  |
| vip_address         | 172.30.107.36                        |
| vip_network_id      | 755e0bef-ad65-46ac-abac-3e183daf050e |
| vip_port_id         | 16824d3b-e9ae-4e73-95c0-e712502c9079 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | acc4583a-e935-4a2b-a06e-db1f3f4a77f5 |
+---------------------+--------------------------------------+


# Removing the service removes the loadbalancer:

$ oc delete service/my-service
service "my-service" deleted
$ openstack loadbalancer show demo/my-service
Unable to locate demo/my-service in loadbalancers
Comment 13 errata-xmlrpc 2021-10-18 17:29:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759