Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2081821

Summary: SpecialResourceModule CRD is not installed after deploying SRO operator using brew bundle image via OLM
Product: OpenShift Container Platform Reporter: yliu1
Component: Special Resource OperatorAssignee: Yoni Bettan <ybettan>
Status: CLOSED ERRATA QA Contact: Constantin Vultur <cvultur>
Severity: high Docs Contact:
Priority: high    
Version: 4.11CC: bzvonar, cvultur, ukalifon, ybettan
Target Milestone: ---Flags: bthurber: needinfo-
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-10 11:10:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description yliu1 2022-05-04 18:07:22 UTC
Description of problem:

After SRO operator is deployed via OLM using brew bundle image, the SpecialResourceModule CRD is missing, and SRO pod keeps restarting.


Version-Release number of selected component (if applicable):
4.11

How reproducible:
Always

Steps to Reproduce:
1. Create an index image from 4.11 brew bundle build:
opm index add  --bundles registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-special-resource-operator-bundle@sha256:5af9394331d29f60b48929e44fd484eb5e357896f944ef5ad33938a1ff9335c1  --tag registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000/sro/sro-index:v4.11

bundle image is from here:
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1998026

2. Use above image to create catalog source on cluster
3. Create namespace, operatorgroup and subscriptions(using the the created catalogsource in step2)
4. Wait for SRO csv to be installed and pod to be running.

Actual results:
SRO operator is deployed successfully, but SpecialResourceModule and PreflightValidation CRDs are not installed and SRO pod keep restarting.

Only SpecialResource CRD gets installed:
[kni@registry ~]$ oc get crd -A | grep -i special
specialresources.sro.openshift.io                                        2022-05-04T15:54:54Z



Expected results:
SpecialResourceModule CRD is installed and SRO pod is running stably.

Additional info:

SRO pod logs ("manager" container):

{"level":"error","ts":1651686626.8944268,"logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":"PreflightValidation.sro.openshift.io","error":"no matches for kind \"PreflightValidation\" in version \"sro.openshift.io/v1beta1\"","stacktrace":"k8s.io/apimachinery/pkg/util/wait.poll\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:591\nk8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:542\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/source/source.go:131"}
{"level":"error","ts":1651686632.9288297,"logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":"SpecialResourceModule.sro.openshift.io","error":"no matches for kind \"SpecialResourceModule\" in version \"sro.openshift.io/v1beta1\"","stacktrace":"k8s.io/apimachinery/pkg/util/wait.poll\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:591\nk8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:542\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/source/source.go:131"}

Comment 6 yliu1 2022-05-31 16:04:14 UTC
Same issue was encountered with 4.11.0-202205301910 SRO brew bundle build.
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2031913

# stable channel was used in subs. 
# installed csv:
openshift-special-resource-operator                openshift-special-resource-operator.4.11.0-202205301910   Special Resource Operator                    4.11.0-202205301910                                        Succeeded

# crds:
[kni@provisionhost-0-0 ~]$ oc get crd |grep special
specialresources.sro.openshift.io                                        2022-05-31T15:52:34Z

# pod keeps restarting
# pod logs:
I0531 15:54:00.567114       1 request.go:665] Waited for 1.742550842s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/security.internal.openshift.io/v1?timeout=32s
{"level":"error","ts":1654012443.2772105,"logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":"SpecialResourceModule.sro.openshift.io","error":"no matches for kind \"SpecialResourceModule\" in version \"sro.openshift.io/v1beta1\"","stacktrace":"k8s.io/apimachinery/pkg/util/wait.poll\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:591\nk8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:542\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/source/source.go:131"}
{"level":"error","ts":1654012449.2234623,"logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":"PreflightValidation.sro.openshift.io","error":"no matches for kind \"PreflightValidation\" in version \"sro.openshift.io/v1beta1\"","stacktrace":"k8s.io/apimachinery/pkg/util/wait.poll\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:591\nk8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:542\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/source/source.go:131"}

Comment 8 yliu1 2022-06-13 18:21:13 UTC
Verified with latest SRO brew build. Expected CRDs are created.

[kni sro-manifests]$ oc get csv -n openshift-special-resource-operator
NAME                                                      DISPLAY                                                                                                                 VERSION               REPLACES   PHASE
openshift-gitops-operator.v1.5.2                          Red Hat OpenShift GitOps                                                                                                1.5.2                            Succeeded
openshift-special-resource-operator.4.11.0-202206130817   Special Resource Operator                                                                                               4.11.0-202206130817              Succeeded
topology-aware-lifecycle-operator.v4.10.0                 Topology Aware Lifecycle Operator is an operator that facilitates platform and operator upgrades of group of clusters   4.10.0                           Succeeded

[kni sro-manifests]$ oc get crd | grep spe
specialresourcemodules.sro.openshift.io                                  2022-06-13T18:14:22Z
specialresources.sro.openshift.io                                        2022-06-13T18:14:22Z
[kni sro-manifests]$ oc get crd | grep -i PreflightValidation
preflightvalidations.sro.openshift.io                                    2022-06-13T18:14:22Z


Procedures used to deploy SRO operator from brew bundle build:
--------------------------------------------------------------
SRO brew build:https://brewweb.engineering.redhat.com/brew/packageinfo?packageID=77344

reg=registry.kni-qe-17.lab.eng.rdu2.redhat.com:5000
dest_index=${reg}/sro/sro-index:v4.11

opm index add --bundles registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-special-resource-operator-bundle@sha256:f34f0f3535c84cc0b23dc59e213a9decabf94313a1fd7a7bd1d99466eb038407 --tag ${dest_index}

podman push ${dest_index}

oc adm catalog mirror ${dest_index} ${reg} -a /home/kni/mirror-combined-secret.json --manifests-only=true --icsp-scope registry --to-manifests=sro-manifests --filter-by-os=linux/amd64

cp sro-manifests/mapping.txt sro-manifests/mapping-clone.txt
sed -i 's|^registry.redhat.io/|brew.registry.redhat.io/|g' sro-manifests/mapping-clone.txt
# remove the index image line from mapping-clone.txt
oc image mirror -f sro-manifests/mapping-clone.txt -a /home/kni/mirror-combined-secret.json '--filter-by-os=.*' --keep-manifest-list --continue-on-error=true 

# At this point the operator images should be successfully mirrored to disconnected registry. 
cd sro-manifests; oc apply -f catalogSource.yaml

Now follow deployment procedures in Deployment doc to deploy the operator: https://docs.openshift.com/container-platform/4.10/hardware_enablement/psap-special-resource-operator.html. Make sure to change the subs yaml to use the catalog source in catalogSource.yaml.

Comment 10 errata-xmlrpc 2022-08-10 11:10:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069