Description of problem:

This BZ has been filed to track the LB type svcs and routes functionality when using the OVN Octavia provider,
it's a test only BZ that needs to be verified once the depending BZs are fixed.

It depends on the bug 2042976 and on the bug 2055629 in OSP python-networking-ovn component, so the functionality
will be broken in 4.11 when using external CCM until those BZs are fixed in OSP.

Version-Release number of selected component (if applicable):
4.11.0-0.nightly-2022-05-05-015322 with eCCM as TP

How reproducible: always


Steps to Reproduce:
1. Install 4.11 with ExternalCloudProvider (no need to do it when it becomes by default)

      $ openshift-install create manifests --log-level=debug --dir=/home/stack/ostest/
      $ cd ostest/
      $ cat <<EOF >manifests/manifest_feature_gate.yaml
      apiVersion: config.openshift.io/v1
      kind: FeatureGate
      metadata:
        annotations:
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
          release.openshift.io/create-only: "true"
        name: cluster
      spec:
        customNoUpgrade:
          enabled:
          - ExternalCloudProvider
        featureSet: CustomNoUpgrade
      EOF

      $ openshift-install create cluster --log-level=debug --dir=/home/stack/ostest/

2. Change the cloud provider Octavia config in order to use the OVN Octavia driver

$ oc get cm cloud-provider-config -n openshift-config -o yaml                                                                                                                                             
[...]
  config: |
    [Global]
    secret-name = openstack-credentials
    secret-namespace = kube-system
    ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem
    [LoadBalancer]
    use-octavia = True
    lb-provider = ovn <-------
    lb-method = SOURCE_IP_PORT <----------
kind: ConfigMap
[...]

The nodes are going to unschedulable and then back to ready, so the change is supposedly applied.

$ oc get nodes
NAME                          STATUS   ROLES    AGE     VERSION
ostest-ffjcv-master-0         Ready    master   4h46m   v1.23.3+54654d2
ostest-ffjcv-master-1         Ready    master   4h45m   v1.23.3+54654d2
ostest-ffjcv-master-2         Ready    master   4h37m   v1.23.3+54654d2
ostest-ffjcv-worker-0-7bwf8   Ready    worker   4h26m   v1.23.3+54654d2
ostest-ffjcv-worker-0-dbj9s   Ready    worker   4h26m   v1.23.3+54654d2
ostest-ffjcv-worker-0-gzjf2   Ready    worker   4h26m   v1.23.3+54654d2


3. Create the loadbalancer type svc with below manifest:

cat <<EOF | oc apply -f -
---
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  name: lb-test-ns
  labels:
    kubernetes.io/metadata.name: lb-test-ns
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lb-test-dep
  namespace: lb-test-ns
  labels:
    app: lb-test-dep
spec:
  replicas: 2
  selector:
    matchLabels:
      app: lb-test-dep
  template:
    metadata:
      labels:
        app: lb-test-dep
    spec:
      containers:
      - image: quay.io/kuryr/demo
        name: demo
---
apiVersion: v1
kind: Service
metadata:
  name: lb-test-svc
  namespace: lb-test-ns
  labels:
    app: lb-test-dep
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: lb-test-dep
  type: LoadBalancer
EOF

4. Check LB, pod and svc creation

LB
--
| 8d001d70-e891-4379-a850-2335819aa7cd | kube_service_kubernetes_lb-test-ns_lb-test-svc | a64676dfa4b24cc9adfb620fef7b6506 | 10.196.3.144 | ACTIVE              | ovn      |                                                                 


Pods
----
lb-test-ns                                         lb-test-dep-68d6754b4d-mjkkh                                1/1     Running     0               153m
lb-test-ns                                         lb-test-dep-68d6754b4d-x47fh                                1/1     Running     0               153m

svc
---
lb-test-ns    lb-test-svc            LoadBalancer   172.30.131.186   10.46.22.227      80:32383/TCP          154m                        


5. Check connectivity to the svc
$ curl 10.46.22.227


Actual results:
$ curl 10.46.22.227
(no reply)

Expected results: reply from the svc pods


Additional info:

LB pools are in PENDING_CREATE status

$ openstack loadbalancer pool list
+--------------------------------------+-------------+----------------------------------+---------------------+----------+----------------+----------------+                                                                                 
| id                                   | name        | project_id                       | provisioning_status | protocol | lb_algorithm   | admin_state_up |                                                                                 
+--------------------------------------+-------------+----------------------------------+---------------------+----------+----------------+----------------+                                                                                 
| 01c61f45-582d-42d3-b40e-ebb7537eb054 | TCP_80_pool | 42a001f17da442e9a0e894a1a7052603 | PENDING_CREATE      | TCP      | SOURCE_IP_PORT | True           |                                                                                 
| 8e62fce6-c544-4f5d-b6eb-519be6a9ace4 | TCP_80_pool | 42a001f17da442e9a0e894a1a7052603 | PENDING_CREATE      | TCP      | SOURCE_IP_PORT | True           |                                                                                 
+--------------------------------------+-------------+----------------------------------+---------------------+----------+----------------+----------------+

$ openstack loadbalancer list
+--------------------------------------+------------------------------------------------+----------------------------------+-------------+---------------------+----------+
| id                                   | name                                           | project_id                       | vip_address | provisioning_status | provider |
+--------------------------------------+------------------------------------------------+----------------------------------+-------------+---------------------+----------+
| bda68836-fea7-4827-9a18-f3a140e3c24f | kube_service_kubernetes_lb-test-ns_lb-test-svc | 42a001f17da442e9a0e894a1a7052603 | 10.196.1.19 | ACTIVE              | ovn      |
+--------------------------------------+------------------------------------------------+----------------------------------+-------------+---------------------+----------+


Issue in cloud-provider-openstack: https://github.com/kubernetes/cloud-provider-openstack/issues/1751