Bug 2209622

Summary: [Fussion-aaS] Agent deployment on Privatelink setup failed for provider and mons, osds are not created
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: suchita <sgatfane>
Component: odf-managed-serviceAssignee: Ohad <omitrani>
Status: NEW --- QA Contact: Neha Berry <nberry>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.12CC: odf-bz-bot
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description suchita 2023-05-24 09:34:07 UTC
Description of problem:
Agent deployment on Privatelink setup failed for provider and mons, osds are not created
rook-ceph-tools stuck in ContainerCreating

Version-Release number of selected component (if applicable):
catsrc image: image: quay.io/rhceph-dev/ocs-registry:4.12.3-17

$ oc get csv ocs-operator.v4.12.3-rhodf -o yaml | grep full
    full_version: 4.12.3-17
ROSA/OCP  4.12.16 


How reproducible:
3/3


Steps to Reproduce:
1. Deploy ROSA Privatelink cluster
2. Deploy agent with image of 4.12.3-17 or 4.12.3-16
3. 

Actual results:
rook-ceph-tools stuck in ContainerCreating state, 
mons, osds are not created

Expected results:
All pods should be created and csv should be succeeded state 

Additional info:
$ oc get pods
NAME                                                              READY   STATUS              RESTARTS   AGE
4261301e16ee6ab3d8f05ac122f9f46dd6d6b340676b8aeb8a8c49a9d7qkw7f   0/1     Completed           0          4m26s
managed-fusion-offering-catalog-z7jsx                             1/1     Running             0          4m57s
ocs-metrics-exporter-8d85ddb7f-dkcjq                              1/1     Running             0          3m45s
ocs-operator-d9b889d-f7rlt                                        1/1     Running             0          3m46s
rook-ceph-operator-6df5dcc59-fdvxr                                1/1     Running             0          3m46s
rook-ceph-tools-774c9c597c-4gqrl                                  0/1     ContainerCreating   0          3m34s


$ oc get service
NAME                              TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE
managed-fusion-offering-catalog   ClusterIP      172.30.165.125   <none>        50051/TCP         11m
ocs-provider-server               LoadBalancer   172.30.126.91    <pending>     50051:31659/TCP   9m40s

$ oc describe service ocs-provider-server
Name:                     ocs-provider-server
Namespace:                fusion-storage
Labels:                   <none>
Annotations:              service.beta.openshift.io/serving-cert-secret-name: ocs-provider-server-cert
Selector:                 app=ocsProviderApiServer
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       172.30.126.91
IPs:                      172.30.126.91
Port:                     <unset>  50051/TCP
TargetPort:               ocs-provider/TCP
NodePort:                 <unset>  31659/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                  Age                  From                Message
  ----     ------                  ----                 ----                -------
  Normal   EnsuringLoadBalancer    4m52s (x7 over 10m)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  4m52s (x7 over 10m)  service-controller  Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB