Description of problem: For selectorless services - i.e. the ones when it's user's responsibility to maintain the Endpoints object Kuryr is not reacting to the deletion of the Endpoints objects and not removing the LB members when it is gone. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Create a Service without selectors. 2. Create some kuryr/demo pod. 3. Create an Endpoints object related to that Service and put IP of the pod from #2 there. 4. Check connectivity to the service. It should work as Kuryr should create an LB related to that Service and put pod from #2 as a member. 5. Delete the Endpoints object, wait ~1 minute. 6. Test connectivity to the Service. Actual results: The service is reachable. Expected results: The service should be not reachable, the LB should have no members or be deleted altogether. Additional info:
Verified on 4.9.0-0.nightly-2021-08-31-123131 on top of OSP16.1 (RHOS-16.1-RHEL-8-20210604.n.0) with OVN-Octavia. # Given below project: $ oc get all -n demo -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/demo-7897db69cc-hrgzb 1/1 Running 0 20h 10.128.130.64 ostest-87ddk-worker-0-2bhm6 <none> <none> pod/demo-7897db69cc-l826k 1/1 Running 0 20h 10.128.130.57 ostest-87ddk-worker-0-2bhm6 <none> <none> pod/demo-7897db69cc-pp4jj 1/1 Running 0 20h 10.128.130.212 ostest-87ddk-worker-0-rsd89 <none> <none> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/my-service ClusterIP 172.30.107.36 <none> 8080/TCP 81s <none> NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/demo 3/3 3 3 20h demo quay.io/kuryr/demo app=demo NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/demo-7897db69cc 3 3 3 20h demo quay.io/kuryr/demo app=demo,pod-template-hash=7897db69cc # A service without selector is created: $ cat service_wo_selectors.yaml apiVersion: v1 kind: Service metadata: name: my-service namespace: demo spec: ports: - protocol: TCP port: 8080 targetPort: 8080 $ oc apply -f service_wo_selectors.yaml service/my-service created # No lb is created on openstack as there are no endpoints: $ openstack loadbalancer show demo/my-service Unable to locate demo/my-service in loadbalancers # Endpoints with the specific PODS IPs is added to my-service service: $ cat endpoints.yaml apiVersion: v1 kind: Endpoints metadata: name: my-service namespace: demo subsets: - addresses: - ip: 10.128.130.64 - ip: 10.128.130.57 - ip: 10.128.130.212 ports: - port: 8080 (shiftstack) [stack@undercloud-0 BZ1938282]$ oc apply -f endpoints.yaml endpoints/my-service created # The lb is created on openstack with below endpoints: $ openstack loadbalancer show demo/my-service +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2021-09-02T10:54:15 | | description | | | flavor_id | None | | id | 54b3ec22-66a3-4eef-a7d1-d221ae576512 | | listeners | d22fcb0f-5eaf-41e0-9257-39ca8cee4275 | | name | demo/my-service | | operating_status | ONLINE | | pools | 3d512969-2b26-4dd9-9f30-17cbbb645033 | | project_id | 8493e10827254256b31b4b5ecefc8420 | | provider | ovn | | provisioning_status | ACTIVE | | updated_at | 2021-09-02T10:54:49 | | vip_address | 172.30.107.36 | | vip_network_id | 755e0bef-ad65-46ac-abac-3e183daf050e | | vip_port_id | 16824d3b-e9ae-4e73-95c0-e712502c9079 | | vip_qos_policy_id | None | | vip_subnet_id | acc4583a-e935-4a2b-a06e-db1f3f4a77f5 | +---------------------+--------------------------------------+ $ openstack loadbalancer member list $(openstack loadbalancer show demo/my-service -c pools -f value) +--------------------------------------+----------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+----------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+ | 0e73175a-5794-489e-ac34-3544e565b8c5 | demo/my-service:8080 | 8493e10827254256b31b4b5ecefc8420 | ACTIVE | 10.128.130.64 | 8080 | NO_MONITOR | 1 | | d0078a7d-0f22-4814-aa96-7b91d5c9e0dc | demo/my-service:8080 | 8493e10827254256b31b4b5ecefc8420 | ACTIVE | 10.128.130.57 | 8080 | NO_MONITOR | 1 | | 60ae48b2-b608-4241-96c7-4fffe503f455 | demo/my-service:8080 | 8493e10827254256b31b4b5ecefc8420 | ACTIVE | 10.128.130.212 | 8080 | NO_MONITOR | 1 | +--------------------------------------+----------------------+----------------------------------+---------------------+----------------+---------------+------------------+--------+ # The svc is working as expected: $ oc rsh demo-7897db69cc-hrgzb curl 172.30.107.36:8080 demo-7897db69cc-hrgzb: HELLO! I AM ALIVE!!! $ oc rsh demo-7897db69cc-hrgzb curl 172.30.107.36:8080 demo-7897db69cc-l826k: HELLO! I AM ALIVE!!! # Removing the enpoints object on OCP removes the members but not the loadbalancer. $ oc delete endpoints my-service endpoints "my-service" deleted $ openstack loadbalancer member list $(openstack loadbalancer show demo/my-service -c pools -f value) $ openstack loadbalancer show demo/my-service +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2021-09-02T10:54:15 | | description | | | flavor_id | None | | id | 54b3ec22-66a3-4eef-a7d1-d221ae576512 | | listeners | d22fcb0f-5eaf-41e0-9257-39ca8cee4275 | | name | demo/my-service | | operating_status | ONLINE | | pools | 3d512969-2b26-4dd9-9f30-17cbbb645033 | | project_id | 8493e10827254256b31b4b5ecefc8420 | | provider | ovn | | provisioning_status | ACTIVE | | updated_at | 2021-09-02T10:58:17 | | vip_address | 172.30.107.36 | | vip_network_id | 755e0bef-ad65-46ac-abac-3e183daf050e | | vip_port_id | 16824d3b-e9ae-4e73-95c0-e712502c9079 | | vip_qos_policy_id | None | | vip_subnet_id | acc4583a-e935-4a2b-a06e-db1f3f4a77f5 | +---------------------+--------------------------------------+ # Removing the service removes the loadbalancer: $ oc delete service/my-service service "my-service" deleted $ openstack loadbalancer show demo/my-service Unable to locate demo/my-service in loadbalancers
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759