Description of problem:

Cannot install openshift v3.11.286 (or v3.11.287) as kuryr-controller pod crashloops due to some incopatibility with Octavia API.
The playbook fails in the task "Wait for the ServiceMonitor CRD to be created".

TASK [openshift_cluster_monitoring_operator : Wait for the ServiceMonitor CRD to be created] ***
FAILED - RETRYING: Wait for the ServiceMonitor CRD to be created (30 retries left).
...
FAILED - RETRYING: Wait for the ServiceMonitor CRD to be created (1 retries left).
fatal: [master-1.openshift.example.com]: FAILED! => {"attempts": 30, "changed": true, "cmd": ["oc", "get", "crd", "servicemonitors.monitoring.coreos.com", "-n", "openshift-monitoring", "--config=/tmp/openshift-cluster-monitoring-ansible-c8Hlpm/admin.kubeconfig"], "delta": "0:00:00.212646", "end": "2020-09-16 12:26:47.048829", "msg": "non-zero return code", "rc": 1, "start": "2020-09-16 12:26:46.836183", "stderr": "No resources found.\nError from server (NotFound): customresourcedefinitions.apiextensions.k8s.io \"servicemonitors.monitoring.coreos.com\" not found", "stderr_lines": ["No resources found.", "Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io \"servicemonitors.monitoring.coreos.com\" not found"], "stdout": "", "stdout_lines": []}

Failure summary:


  1. Hosts:    master-1.openshift.example.com
     Play:     Configure Cluster Monitoring Operator
     Task:     Wait for the ServiceMonitor CRD to be created
     Message:  non-zero return code


[openshift@master-0 ~]$ oc -n kuryr get pods
NAME                                READY     STATUS             RESTARTS   AGE
kuryr-cni-ds-25rcs                  2/2       Running            90         14h
kuryr-cni-ds-2v9c2                  2/2       Running            0          14h
kuryr-cni-ds-84kxq                  2/2       Running            0          14h
kuryr-cni-ds-8n7mb                  2/2       Running            28         14h
kuryr-cni-ds-8qcv5                  2/2       Running            0          14h
kuryr-cni-ds-9xds6                  2/2       Running            0          14h
kuryr-cni-ds-np5wr                  2/2       Running            0          14h
kuryr-cni-ds-x97jv                  2/2       Running            28         14h
kuryr-controller-6dd96d9587-jzp92   0/1       CrashLoopBackOff   169        14h


kuryr-controller pod logs:
2020-09-17 06:11:25.793 1 INFO kuryr_kubernetes.config [-] /usr/bin/kuryr-k8s-controller version 0.0.0
2020-09-17 06:11:25.956 1 INFO os_vif [-] Loaded VIF plugins: noop, sriov, ovs, linux_bridge, noop
2020-09-17 06:11:25.959 1 INFO kuryr_kubernetes.controller.service [-] Configured handlers: ['vif', 'lb', 'lbaasspec', 'namespace']
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service [-] Exception when loading handlers lb = kuryr_kubernetes.controller.handlers.lbaas:LoadBalancerHandler.: AttributeError: 'Proxy' object has no attribute 'get_all_version_data'
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service Traceback (most recent call last):
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 195, in _load_plugins
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     verify_requirements,
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 158, in _load_one_plugin
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     verify_requirements,
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 227, in _load_one_plugin
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     obj = plugin(*invoke_args, **invoke_kwds)
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 162, in __init__
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     self._drv_lbaas = drv_base.LBaaSDriver.get_instance()
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/base.py", line 80, in get_instance
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     invoke_on_load=True)
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 61, in __init__
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     warn_on_missing_entrypoint=warn_on_missing_entrypoint
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 81, in __init__
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     verify_requirements)
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 203, in _load_plugins
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     self._on_load_failure_callback(self, ep, err)
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 195, in _load_plugins
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     verify_requirements,
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 158, in _load_one_plugin
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     verify_requirements,
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 227, in _load_one_plugin
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     obj = plugin(*invoke_args, **invoke_kwds)
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/lbaasv2.py", line 76, in __init__
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     v = self.get_octavia_version()
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/lbaasv2.py", line 101, in get_octavia_version
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service     regions = lbaas.get_all_version_data()
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service AttributeError: 'Proxy' object has no attribute 'get_all_version_data'
2020-09-17 06:11:25.977 1 ERROR kuryr_kubernetes.controller.service 
2020-09-17 06:11:25.979 1 CRITICAL kuryr_kubernetes.controller.service [-] Handlers entrypoint "lb = kuryr_kubernetes.controller.handlers.lbaas:LoadBalancerHandler" failed to load due to 'Proxy' object has no attribute 'get_all_version_data'.


Version-Release number of selected component (if applicable):
openshift v3.11.286
OSP 13 2020-09-03.2
kuryr controller image: https://access.redhat.com/containers/#/registry.access.redhat.com/openshift3/ose-kuryr-controller/images/v3.11.286-1
kuryr cni image: https://access.redhat.com/containers/#/registry.access.redhat.com/openshift3/ose-kuryr-cni/images/v3.11.286-1

How reproducible: always


Steps to Reproduce:
1. Deploy OSP 13 with octavia
2. Install 3.11 with Kuryr

Actual results: Installer fails 


Expected results: Successful installation