Description of problem:

Namespace deletion remains in terminating status when it holds a LoadBalancer type svc, which is
not deleted either.

$ oc get project
NAME                                               DISPLAY NAME   STATUS
[...]
lb-test-ns                                                        Terminating

$ oc get svc -n lb-test-ns lb-test-svc
NAME          TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)        AGE
lb-test-svc   LoadBalancer   172.30.238.135   10.46.22.247   80:32469/TCP   24m


Version-Release number of selected component (if applicable):
OCP 4.11.0-0.nightly-2022-04-08-205307
OSP 16.1.7


How reproducible: always


Steps to Reproduce:
1. Install 4.11 with ExternalCloudProvider

      $ openshift-install create manifests --log-level=debug --dir=/home/stack/ostest/
      $ cd ostest/
      $ cat <<EOF >manifests/manifest_feature_gate.yaml
      apiVersion: config.openshift.io/v1
      kind: FeatureGate
      metadata:
        annotations:
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
          release.openshift.io/create-only: "true"
        name: cluster
      spec:
        customNoUpgrade:
          enabled:
          - ExternalCloudProvider
        featureSet: CustomNoUpgrade
      EOF

      $ openshift-install create cluster --log-level=debug --dir=/home/stack/ostest/

2. Create a ns, deployment and LB type svc

$ cat lb_test_resources.yaml
---                                                                                                                                                                                                                                           
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  name: lb-test-ns
  labels:
    kubernetes.io/metadata.name: lb-test-ns
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lb-test-dep
  namespace: lb-test-ns
  labels:
    app: lb-test-dep
spec:
  replicas: 2
  selector:
    matchLabels:
      app: lb-test-dep
  template:
    metadata:
      labels:
        app: lb-test-dep
    spec:
      containers:
      - image: quay.io/kuryr/demo
        name: demo
---
apiVersion: v1
kind: Service
metadata:
  name: lb-test-svc
  namespace: lb-test-ns
  labels:
    app: lb-test-dep
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: lb-test-dep
  type: LoadBalancer


      $ oc apply -f lb_test_resources.yaml

3. Wait until the svc is assigned a fip:

      $ oc get svc -n lb-test-ns lb-test-svc
      NAME          TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)        AGE
      lb-test-svc   LoadBalancer   172.30.238.135   10.46.22.247   80:32469/TCP   3

4. Check the connectivity to the svc:

      $ curl 10.46.22.247                                                                                                                                                                           
      lb-test-dep-68d6754b4d-b9lmp: HELLO! I AM ALIVE!!!                                                                                                                                                                  
      $ curl 10.46.22.247                                                                                                                                                                           
      lb-test-dep-68d6754b4d-lxpdg: HELLO! I AM ALIVE!!!  

5. Delete the ns:

      $ oc delete project lb-test-ns                                                                                                                                                                
      project.project.openshift.io "lb-test-ns" deleted

6. Check the ns is deleted

Actual results: the namespace (and the LB type svc) are not deleted

$ oc get project
NAME                                               DISPLAY NAME   STATUS
[...]
lb-test-ns                                                        Terminating


Expected results: Namespace (and LB type svc) deleted


Additional info:

$ oc get svc -n lb-test-ns lb-test-svc -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"lb-test-dep"},"name":"lb-test-svc","namespace":"lb-test-ns"},"spec":{"ports":[{"port":80,"targetPort":8080}],"selector":{"app":"lb-test-dep"},"type":"LoadBalancer"}}
  creationTimestamp: "2022-04-12T09:00:41Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2022-04-12T09:02:36Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app: lb-test-dep
  name: lb-test-svc
  namespace: lb-test-ns
  resourceVersion: "64754"
  uid: 01559d25-9e4a-43c8-9e81-9343f7eaa500
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 172.30.238.135
  clusterIPs:
  - 172.30.238.135
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - nodePort: 32469
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: lb-test-dep
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 10.46.22.247

$ oc describe svc -n lb-test-ns lb-test-svc
Name:                     lb-test-svc
Namespace:                lb-test-ns
Labels:                   app=lb-test-dep
Annotations:              <none>
Selector:                 app=lb-test-dep
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       172.30.238.135
IPs:                      172.30.238.135
LoadBalancer Ingress:     10.46.22.247
Port:                     <unset>  80/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  32469/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>