Description of problem: The compliance-operator installation fails dues to secret "compliance-operator-serving-cert" not found $ oc get csv -nopenshift-compliance NAME DISPLAY VERSION REPLACES PHASE compliance-operator.v0.1.37 Compliance Operator 0.1.37 Installing $ oc get pods -nopenshift-compliance NAME READY STATUS RESTARTS AGE compliance-operator-646dd4bd57-st6nh 0/1 ContainerCreating 0 6m33s $ oc describe csv compliance-operator.v0.1.37 -nopenshift-compliance |tail -5 Normal NeedsReinstall 2m (x2 over 2m1s) operator-lifecycle-manager installing: waiting for deployment compliance-operator to become ready: deployment "compliance-operator" not available: Deployment does not have minimum availability. Normal AllRequirementsMet 119s (x4 over 7m2s) operator-lifecycle-manager all requirements found, attempting install Normal InstallSucceeded 119s (x4 over 7m1s) operator-lifecycle-manager waiting for install components to report healthy Normal InstallWaiting 119s operator-lifecycle-manager installing: waiting for deployment compliance-operator to become ready: waiting for spec update of deployment "compliance-operator" to be observed... Normal InstallWaiting 118s (x5 over 7m1s) operator-lifecycle-manager installing: waiting for deployment compliance-operator to become ready: deployment "compliance-operator" not available: Deployment does not have minimum availability. $ oc describe pod compliance-operator-646dd4bd57-st6nh -nopenshift-compliance |tail -5 ---- ------ ---- ---- ------- Normal Scheduled 7m41s default-scheduler Successfully assigned openshift-compliance/compliance-operator-646dd4bd57-st6nh to ip-10-0-152-232.us-east-2.compute.internal Warning FailedMount 5m38s kubelet Unable to attach or mount volumes: unmounted volumes=[serving-cert], unattached volumes=[kube-api-access-wjdl6 serving-cert]: timed out waiting for the condition Warning FailedMount 89s (x11 over 7m41s) kubelet MountVolume.SetUp failed for volume "serving-cert" : secret "compliance-operator-serving-cert" not found Warning FailedMount 65s (x2 over 3m23s) kubelet Unable to attach or mount volumes: unmounted volumes=[serving-cert], unattached volumes=[serving-cert kube-api-access-wjdl6]: timed out waiting for the condition ]$ oc get events -nopenshift-compliance LAST SEEN TYPE REASON OBJECT MESSAGE 7m52s Normal Scheduled pod/compliance-operator-646dd4bd57-st6nh Successfully assigned openshift-compliance/compliance-operator-646dd4bd57-st6nh to ip-10-0-152-232.us-east-2.compute.internal 101s Warning FailedMount pod/compliance-operator-646dd4bd57-st6nh MountVolume.SetUp failed for volume "serving-cert" : secret "compliance-operator-serving-cert" not found 5m50s Warning FailedMount pod/compliance-operator-646dd4bd57-st6nh Unable to attach or mount volumes: unmounted volumes=[serving-cert], unattached volumes=[kube-api-access-wjdl6 serving-cert]: timed out waiting for the condition 77s Warning FailedMount pod/compliance-operator-646dd4bd57-st6nh Unable to attach or mount volumes: unmounted volumes=[serving-cert], unattached volumes=[serving-cert kube-api-access-wjdl6]: timed out waiting for the condition 7m53s Normal SuccessfulCreate replicaset/compliance-operator-646dd4bd57 Created pod: compliance-operator-646dd4bd57-st6nh 7m53s Normal ScalingReplicaSet deployment/compliance-operator Scaled up replica set compliance-operator-646dd4bd57 to 1 8m2s Normal RequirementsUnknown clusterserviceversion/compliance-operator.v0.1.37 requirements not yet checked 7m59s Normal RequirementsNotMet clusterserviceversion/compliance-operator.v0.1.37 one or more requirements couldn't be found 2m51s Normal AllRequirementsMet clusterserviceversion/compliance-operator.v0.1.37 all requirements found, attempting install 2m51s Normal InstallSucceeded clusterserviceversion/compliance-operator.v0.1.37 waiting for install components to report healthy 2m50s Normal InstallWaiting clusterserviceversion/compliance-operator.v0.1.37 installing: waiting for deployment compliance-operator to become ready: deployment "compliance-operator" not available: Deployment does not have minimum availability. 2m53s Warning InstallCheckFailed clusterserviceversion/compliance-operator.v0.1.37 install timeout 2m52s Normal NeedsReinstall clusterserviceversion/compliance-operator.v0.1.37 installing: waiting for deployment compliance-operator to become ready: deployment "compliance-operator" not available: Deployment does not have minimum availability. 2m51s Normal InstallWaiting clusterserviceversion/compliance-operator.v0.1.37 installing: waiting for deployment compliance-operator to become ready: waiting for spec update of deployment "compliance-operator" to be observed... Version-Release number of selected component (if applicable): 4.9.0-0.nightly-2021-08-04-131508 + compliance-operator.v0.1.37 How reproducible: Always Steps to Reproduce: 1. Deploy compliance operator upstream/ downstream 2. Check CSV it stuck in installing phase $ oc get csv -nopenshift-compliance 3. Compliance operator pods stuck in ContainerCreating state $ oc get pods -nopenshift-compliance 4. Describe pods oc describe pod compliance-operator-646dd4bd57-st6nh -nopenshift-compliance |tail -5 Actual results: The compliance operator installation is failing due to a secret "compliance-operator-serving-cert" not found issue. Expected results: The compliance operator should get installed without any secret issue. Additional info: Noticed same issue with downstream compliance operator build https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1686222
I am able to install compliance-operator.v0.1.36 on the same cluster. https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1685104 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-08-04-131508 True False 11h Cluster version is 4.9.0-0.nightly-2021-08-04-131508 $ oc get sub -n openshift-compliance NAME PACKAGE SOURCE CHANNEL openshift-compliance-operator compliance-operator compliance-operator release-0.1 $ oc get csv -n openshift-compliance NAME DISPLAY VERSION REPLACES PHASE compliance-operator.v0.1.36 Compliance Operator 0.1.36 Succeeded $ oc get pods -n openshift-compliance NAME READY STATUS RESTARTS AGE compliance-operator-7b874dc8b5-cpl5c 1/1 Running 0 2m31s ocp4-openshift-compliance-pp-5f4f99b899-6sqjs 1/1 Running 0 98s rhcos4-openshift-compliance-pp-59c49cc655-z6wv5 1/1 Running 0 98s
Hi Juan & Matt, I tried to installed Compliance Operator with latest version v0.1.38 using the latest operator build https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1699927 but it is failing with the below error. $ oc get csv NAME DISPLAY VERSION REPLACES PHASE compliance-operator.v0.1.38 Compliance Operator 0.1.38 Succeeded elasticsearch-operator.5.2.0-36 OpenShift Elasticsearch Operator 5.2.0-36 Succeeded $ oc get pods NAME READY STATUS RESTARTS AGE compliance-operator-77c6d74b99-fvjv5 1/1 Running 1 (6m7s ago) 6m50s ocp4-openshift-compliance-pp-68f698ff67-tj859 0/1 Init:CreateContainerError 0 5m30s rhcos4-openshift-compliance-pp-76d59875bc-qgzg4 0/1 Init:CreateContainerError 0 5m30s $ oc describe pod ocp4-openshift-compliance-pp-68f698ff67-tj859 -nopenshift-compliance |tail Normal Pulled 5m2s kubelet Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.513314997s Normal Pulled 4m45s kubelet Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.434213353s Warning Failed 4m45s kubelet Error: container create failed: time="2021-08-17T04:20:47Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH" Normal Pulled 4m30s kubelet Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.652491468s Warning Failed 4m30s kubelet Error: container create failed: time="2021-08-17T04:21:02Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH" Warning Failed 4m17s kubelet Error: container create failed: time="2021-08-17T04:21:15Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH" Normal Pulled 4m17s kubelet Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.532629606s Normal Pulled 4m2s kubelet Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.454426391s Warning Failed 4m2s kubelet Error: container create failed: time="2021-08-17T04:21:30Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH" Normal Pulling 33s (x22 over 5m40s) kubelet Pulling image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" $ oc describe pod rhcos4-openshift-compliance-pp-76d59875bc-qgzg4 -nopenshift-compliance |tail Normal Pulled 5m18s kubelet Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.576205601s Normal Pulled 5m3s kubelet Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.551628613s Warning Failed 5m2s kubelet Error: container create failed: time="2021-08-17T04:20:45Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH" Normal Pulled 4m48s kubelet Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.569156248s Warning Failed 4m47s kubelet Error: container create failed: time="2021-08-17T04:21:01Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH" Warning Failed 4m33s kubelet Error: container create failed: time="2021-08-17T04:21:15Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH" Normal Pulled 4m33s kubelet Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.560369616s Normal Pulled 4m18s kubelet Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.572337567s Warning Failed 4m17s kubelet Error: container create failed: time="2021-08-17T04:21:31Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH" Normal Pulling 53s (x22 over 5m56s) kubelet Pulling image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" However, I am able to install Compliance Operator with the upstream build $ oc get csv NAME DISPLAY VERSION REPLACES PHASE compliance-operator.v0.1.38 Compliance Operator 0.1.38 Succeeded elasticsearch-operator.5.2.0-36 OpenShift Elasticsearch Operator 5.2.0-36 Succeeded $ oc get pods NAME READY STATUS RESTARTS AGE compliance-operator-d7776cbff-7x6k8 1/1 Running 1 (2m20s ago) 2m57s ocp4-openshift-compliance-pp-5c5c48c759-7t678 1/1 Running 0 104s rhcos4-openshift-compliance-pp-5c7cdf7b67-fz8kl 1/1 Running 0 104s $ oc describe pod rhcos4-openshift-compliance-pp-5c7cdf7b67-fz8kl |tail -15 Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m51s default-scheduler Successfully assigned openshift-compliance/rhcos4-openshift-compliance-pp-5c7cdf7b67-fz8kl to ip-10-0-142-175.us-east-2.compute.internal Normal AddedInterface 3m50s multus Add eth0 [10.129.0.42/23] from openshift-sdn Normal Pulling 3m50s kubelet Pulling image "quay.io/complianceascode/ocp4:latest" Normal Pulled 3m45s kubelet Successfully pulled image "quay.io/complianceascode/ocp4:latest" in 4.729183621s Normal Created 3m44s kubelet Created container content-container Normal Started 3m44s kubelet Started container content-container Normal Pulling 3m43s kubelet Pulling image "quay.io/compliance-operator/compliance-operator:0.1.38" Normal Pulled 3m40s kubelet Successfully pulled image "quay.io/compliance-operator/compliance-operator:0.1.38" in 3.479770454s Normal Created 3m39s kubelet Created container profileparser Normal Started 3m39s kubelet Started container profileparser Normal Pulled 2m36s kubelet Container image "quay.io/compliance-operator/compliance-operator:0.1.38" already present on machine Normal Created 2m35s kubelet Created container pauser Normal Started 2m35s kubelet Started container pauser
[Bug_Verification] I am able to install Compliance Operator with latest version v0.1.39 using the latest build. https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1706646 $ oc project openshift-compliance Now using project "openshift-compliance" on server "https://api.pdhamdhe-2348.qe.devcluster.openshift.com:6443". $ oc get csv NAME DISPLAY VERSION REPLACES PHASE compliance-operator.v0.1.39 Compliance Operator 0.1.39 Succeeded elasticsearch-operator.5.1.1-42 OpenShift Elasticsearch Operator 5.1.1-42 Succeeded $ oc get sub NAME PACKAGE SOURCE CHANNEL compliance-operator compliance-operator compliance-operator release-0.1 $ oc get pods NAME READY STATUS RESTARTS AGE compliance-operator-bb9f644cc-xwfnq 1/1 Running 1 92m ocp4-openshift-compliance-pp-6d7c7db4bd-jwnnq 1/1 Running 0 91m rhcos4-openshift-compliance-pp-c7b548bd-9hqvz 1/1 Running 0 91m $ oc describe pod compliance-operator-bb9f644cc-xwfnq |grep "RELATED_IMAGE" RELATED_IMAGE_OPENSCAP: registry.redhat.io/compliance/openshift-compliance-openscap-rhel8@sha256:1317d69bafd867f358b30168ca40dd49835696821d3f4b1244089f8d46dde3bf RELATED_IMAGE_OPERATOR: registry.redhat.io/compliance/openshift-compliance-rhel8-operator@sha256:10dcd18f4ddf73b295befde8fe720a49d1826d6415b822a4c11a933a8d7a72cb RELATED_IMAGE_PROFILE: registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:7527f52d9ce5f22fe5453b23eb490ce1f7e64e0cf7112b13a294c4bc442ae35d
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Compliance Operator bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3214