Bug 2039373

Summary: [osp][octavia lb] cannot create LoadBalancer type svcs
Product: OpenShift Container Platform Reporter: Martin André <m.andre>
Component: Cloud ComputeAssignee: Martin André <m.andre>
Cloud Compute sub component: OpenStack Provider QA Contact: Jon Uriarte <juriarte>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: andcosta, aos-bugs, cshepher, emacchi, gferrazs, juriarte, mabajodu, m.andre, mbooth, mdulko, mfedosin, mfojtik, nagrawal, pprinett
Version: 4.8Keywords: Triaged
Target Milestone: ---   
Target Release: 4.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: There's a race condition between OpenStack credentials secret creation and kube-controller-manager startup. Consequence: If it happens OpenStack cloud-provider will not get configured with OpenStack credentials, effectively breaking support for creating Octavia load balancers for LoadBalancer services. Fix: Retry fetching the OpenStack credentials secret until it succeeds during the kube-controller-manager started. Result: The OpenStack cloud-provider is now consistently initialized upon kube-controller-manager startup.
 Story Points: ---
Clone Of: 2004542
: 2039377 (view as bug list) Environment:
Last Closed: 2022-02-14 12:00:47 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2004542    
Bug Blocks: 2039377    

Comment 1 ShiftStack Bugwatcher 2022-01-12 07:03:50 UTC 
Removing the Triaged keyword because:
* the QE automation assessment (flag qe_test_coverage) is missing
Comment 6 Jon Uriarte 2022-02-04 12:18:35 UTC 
Verified on 4.9.0-0.nightly-2022-02-02-193336 on top of OSP 16.1.7.

Tried out 8 times a fresh OCP installation and LB creation worked after it all the times.


Procedure:
1. Install OCP

2. Create a project, deployment and a LB type svc
cat <<EOF | oc apply -f -
---
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  name: lb-test-ns
  labels:
    kubernetes.io/metadata.name: lb-test-ns
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lb-test-dep
  namespace: lb-test-ns
  labels:
    app: lb-test-dep
spec:
  replicas: 2
  selector:
    matchLabels:
      app: lb-test-dep
  template:
    metadata:
      labels:
        app: lb-test-dep
    spec:
      containers:
      - image: quay.io/kuryr/demo
        name: demo
---
apiVersion: v1
kind: Service
metadata:
  name: lb-test-svc
  namespace: lb-test-ns
  labels:
    app: lb-test-dep
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: lb-test-dep
  type: LoadBalancer
EOF

$ oc -n lb-test-ns get pods
NAME                          READY   STATUS    RESTARTS   AGE
lb-test-dep-8476c494b-57ww7   1/1     Running   0          5m7s
lb-test-dep-8476c494b-rbw58   1/1     Running   0          5m7s

3. Check the LB creation in OSP
$ openstack loadbalancer list
+--------------------------------------+----------------------------------+----------------------------------+-------------+---------------------+----------+
| id                                   | name                             | project_id                       | vip_address | provisioning_status | provider |
+--------------------------------------+----------------------------------+----------------------------------+-------------+---------------------+----------+
| d8966c19-851a-40be-95e1-ca248a03ca4e | adbaa1c0dd4fe4c789f2f65fe0ad4b25 | a0062929e9ae42b1b75c6f1b2fbdbd2f | 10.196.0.31 | ACTIVE              | amphora  |
+--------------------------------------+----------------------------------+----------------------------------+-------------+---------------------+----------+

4. Check the svc has been assigned the FIP
$ oc -n lb-test-ns get svc
NAME          TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)        AGE
lb-test-svc   LoadBalancer   172.30.156.158  10.46.22.227   80:32688/TCP   5m57s

5. Test the svc
$ curl 10.46.22.227
lb-test-dep-8476c494b-rbw58: HELLO! I AM ALIVE!!!

$ curl 10.46.22.227
lb-test-dep-8476c494b-57ww7: HELLO! I AM ALIVE!!!
Comment 9 errata-xmlrpc 2022-02-14 12:00:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.21 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0488