Bug 1990836

Summary: [v4.9] The compliance-operator installation fails due to secret "compliance-operator-serving-cert" not found
Product: OpenShift Container Platform Reporter: Prashant Dhamdhere <pdhamdhe>
Component: Compliance OperatorAssignee: Matt Rogers <mrogers>
Status: CLOSED ERRATA QA Contact: Prashant Dhamdhere <pdhamdhe>
Severity: urgent Docs Contact:
Priority: high    
Version: 4.9CC: josorior, mrogers, xiyuan
Target Milestone: ---   
Target Release: 4.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-07 06:05:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1946512, 1975358, 1982142, 1983062, 1983878    

Description Prashant Dhamdhere 2021-08-06 11:43:54 UTC 
Description of problem:
The compliance-operator installation fails dues to secret "compliance-operator-serving-cert" not found

$ oc get csv -nopenshift-compliance
NAME                          DISPLAY               VERSION   REPLACES   PHASE
compliance-operator.v0.1.37   Compliance Operator   0.1.37               Installing

$ oc get pods -nopenshift-compliance
NAME                                   READY   STATUS              RESTARTS   AGE
compliance-operator-646dd4bd57-st6nh   0/1     ContainerCreating   0          6m33s

$ oc describe csv compliance-operator.v0.1.37 -nopenshift-compliance |tail -5
  Normal   NeedsReinstall       2m (x2 over 2m1s)    operator-lifecycle-manager  installing: waiting for deployment compliance-operator to become ready: deployment "compliance-operator" not available: Deployment does not have minimum availability.
  Normal   AllRequirementsMet   119s (x4 over 7m2s)  operator-lifecycle-manager  all requirements found, attempting install
  Normal   InstallSucceeded     119s (x4 over 7m1s)  operator-lifecycle-manager  waiting for install components to report healthy
  Normal   InstallWaiting       119s                 operator-lifecycle-manager  installing: waiting for deployment compliance-operator to become ready: waiting for spec update of deployment "compliance-operator" to be observed...
  Normal   InstallWaiting       118s (x5 over 7m1s)  operator-lifecycle-manager  installing: waiting for deployment compliance-operator to become ready: deployment "compliance-operator" not available: Deployment does not have minimum availability.

$ oc describe pod compliance-operator-646dd4bd57-st6nh -nopenshift-compliance |tail -5
  ----     ------       ----                  ----               -------
  Normal   Scheduled    7m41s                 default-scheduler  Successfully assigned openshift-compliance/compliance-operator-646dd4bd57-st6nh to ip-10-0-152-232.us-east-2.compute.internal
  Warning  FailedMount  5m38s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[serving-cert], unattached volumes=[kube-api-access-wjdl6 serving-cert]: timed out waiting for the condition
  Warning  FailedMount  89s (x11 over 7m41s)  kubelet            MountVolume.SetUp failed for volume "serving-cert" : secret "compliance-operator-serving-cert" not found
  Warning  FailedMount  65s (x2 over 3m23s)   kubelet            Unable to attach or mount volumes: unmounted volumes=[serving-cert], unattached volumes=[serving-cert kube-api-access-wjdl6]: timed out waiting for the condition

]$ oc get events -nopenshift-compliance
LAST SEEN   TYPE      REASON                OBJECT                                              MESSAGE
7m52s       Normal    Scheduled             pod/compliance-operator-646dd4bd57-st6nh            Successfully assigned openshift-compliance/compliance-operator-646dd4bd57-st6nh to ip-10-0-152-232.us-east-2.compute.internal
101s        Warning   FailedMount           pod/compliance-operator-646dd4bd57-st6nh            MountVolume.SetUp failed for volume "serving-cert" : secret "compliance-operator-serving-cert" not found
5m50s       Warning   FailedMount           pod/compliance-operator-646dd4bd57-st6nh            Unable to attach or mount volumes: unmounted volumes=[serving-cert], unattached volumes=[kube-api-access-wjdl6 serving-cert]: timed out waiting for the condition
77s         Warning   FailedMount           pod/compliance-operator-646dd4bd57-st6nh            Unable to attach or mount volumes: unmounted volumes=[serving-cert], unattached volumes=[serving-cert kube-api-access-wjdl6]: timed out waiting for the condition
7m53s       Normal    SuccessfulCreate      replicaset/compliance-operator-646dd4bd57           Created pod: compliance-operator-646dd4bd57-st6nh
7m53s       Normal    ScalingReplicaSet     deployment/compliance-operator                      Scaled up replica set compliance-operator-646dd4bd57 to 1
8m2s        Normal    RequirementsUnknown   clusterserviceversion/compliance-operator.v0.1.37   requirements not yet checked
7m59s       Normal    RequirementsNotMet    clusterserviceversion/compliance-operator.v0.1.37   one or more requirements couldn't be found
2m51s       Normal    AllRequirementsMet    clusterserviceversion/compliance-operator.v0.1.37   all requirements found, attempting install
2m51s       Normal    InstallSucceeded      clusterserviceversion/compliance-operator.v0.1.37   waiting for install components to report healthy
2m50s       Normal    InstallWaiting        clusterserviceversion/compliance-operator.v0.1.37   installing: waiting for deployment compliance-operator to become ready: deployment "compliance-operator" not available: Deployment does not have minimum availability.
2m53s       Warning   InstallCheckFailed    clusterserviceversion/compliance-operator.v0.1.37   install timeout
2m52s       Normal    NeedsReinstall        clusterserviceversion/compliance-operator.v0.1.37   installing: waiting for deployment compliance-operator to become ready: deployment "compliance-operator" not available: Deployment does not have minimum availability.
2m51s       Normal    InstallWaiting        clusterserviceversion/compliance-operator.v0.1.37   installing: waiting for deployment compliance-operator to become ready: waiting for spec update of deployment "compliance-operator" to be observed...


Version-Release number of selected component (if applicable):
4.9.0-0.nightly-2021-08-04-131508 + compliance-operator.v0.1.37

How reproducible:
Always

Steps to Reproduce:

1. Deploy compliance operator upstream/ downstream
2. Check CSV it stuck in installing phase
$ oc get csv -nopenshift-compliance
3. Compliance operator pods stuck in ContainerCreating state
$ oc get pods -nopenshift-compliance
4. Describe pods
oc describe pod compliance-operator-646dd4bd57-st6nh -nopenshift-compliance |tail -5

Actual results:
The compliance operator installation is failing due to a secret "compliance-operator-serving-cert" not found issue.

Expected results:
The compliance operator should get installed without any secret issue.

Additional info:
Noticed same issue with downstream compliance operator build
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1686222
Comment 2 Prashant Dhamdhere 2021-08-06 14:21:14 UTC 
I am able to install compliance-operator.v0.1.36 on the same cluster. 
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1685104

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-08-04-131508   True        False         11h     Cluster version is 4.9.0-0.nightly-2021-08-04-131508

$ oc get sub -n openshift-compliance
NAME                            PACKAGE               SOURCE                CHANNEL
openshift-compliance-operator   compliance-operator   compliance-operator   release-0.1

$ oc get csv -n openshift-compliance
NAME                          DISPLAY               VERSION   REPLACES   PHASE
compliance-operator.v0.1.36   Compliance Operator   0.1.36               Succeeded

$ oc get pods -n openshift-compliance
NAME                                              READY   STATUS    RESTARTS   AGE
compliance-operator-7b874dc8b5-cpl5c              1/1     Running   0          2m31s
ocp4-openshift-compliance-pp-5f4f99b899-6sqjs     1/1     Running   0          98s
rhcos4-openshift-compliance-pp-59c49cc655-z6wv5   1/1     Running   0          98s
Comment 6 Prashant Dhamdhere 2021-08-17 04:45:49 UTC 
Hi Juan & Matt,

I tried to installed Compliance Operator with latest version v0.1.38 using the latest operator build
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1699927 but it is failing with the below 
error.


$ oc get csv
NAME                              DISPLAY                            VERSION    REPLACES   PHASE
compliance-operator.v0.1.38       Compliance Operator                0.1.38                Succeeded
elasticsearch-operator.5.2.0-36   OpenShift Elasticsearch Operator   5.2.0-36              Succeeded

$ oc get pods
NAME                                              READY   STATUS                      RESTARTS       AGE
compliance-operator-77c6d74b99-fvjv5              1/1     Running                     1 (6m7s ago)   6m50s
ocp4-openshift-compliance-pp-68f698ff67-tj859     0/1     Init:CreateContainerError   0              5m30s
rhcos4-openshift-compliance-pp-76d59875bc-qgzg4   0/1     Init:CreateContainerError   0              5m30s

$ oc describe pod ocp4-openshift-compliance-pp-68f698ff67-tj859  -nopenshift-compliance |tail
  Normal   Pulled   5m2s                  kubelet  Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.513314997s
  Normal   Pulled   4m45s                 kubelet  Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.434213353s
  Warning  Failed   4m45s                 kubelet  Error: container create failed: time="2021-08-17T04:20:47Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH"
  Normal   Pulled   4m30s                 kubelet  Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.652491468s
  Warning  Failed   4m30s                 kubelet  Error: container create failed: time="2021-08-17T04:21:02Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH"
  Warning  Failed   4m17s                 kubelet  Error: container create failed: time="2021-08-17T04:21:15Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH"
  Normal   Pulled   4m17s                 kubelet  Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.532629606s
  Normal   Pulled   4m2s                  kubelet  Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.454426391s
  Warning  Failed   4m2s                  kubelet  Error: container create failed: time="2021-08-17T04:21:30Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH"
  Normal   Pulling  33s (x22 over 5m40s)  kubelet  Pulling image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118"

$ oc describe pod rhcos4-openshift-compliance-pp-76d59875bc-qgzg4  -nopenshift-compliance |tail
  Normal   Pulled          5m18s                 kubelet            Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.576205601s
  Normal   Pulled          5m3s                  kubelet            Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.551628613s
  Warning  Failed          5m2s                  kubelet            Error: container create failed: time="2021-08-17T04:20:45Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH"
  Normal   Pulled          4m48s                 kubelet            Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.569156248s
  Warning  Failed          4m47s                 kubelet            Error: container create failed: time="2021-08-17T04:21:01Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH"
  Warning  Failed          4m33s                 kubelet            Error: container create failed: time="2021-08-17T04:21:15Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH"
  Normal   Pulled          4m33s                 kubelet            Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.560369616s
  Normal   Pulled          4m18s                 kubelet            Successfully pulled image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118" in 1.572337567s
  Warning  Failed          4m17s                 kubelet            Error: container create failed: time="2021-08-17T04:21:31Z" level=error msg="container_linux.go:380: starting container process caused: exec: \"sh\": executable file not found in $PATH"
  Normal   Pulling         53s (x22 over 5m56s)  kubelet            Pulling image "registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:a4031e10dcb18e77b97d56011a64c77add37a265d8ae278e2d81b67efca69118"


However, I am able to install Compliance Operator with the upstream build

$ oc get csv
NAME                              DISPLAY                            VERSION    REPLACES   PHASE
compliance-operator.v0.1.38       Compliance Operator                0.1.38                Succeeded
elasticsearch-operator.5.2.0-36   OpenShift Elasticsearch Operator   5.2.0-36              Succeeded

$ oc get pods
NAME                                              READY   STATUS    RESTARTS        AGE
compliance-operator-d7776cbff-7x6k8               1/1     Running   1 (2m20s ago)   2m57s
ocp4-openshift-compliance-pp-5c5c48c759-7t678     1/1     Running   0               104s
rhcos4-openshift-compliance-pp-5c7cdf7b67-fz8kl   1/1     Running   0               104s

$ oc describe pod rhcos4-openshift-compliance-pp-5c7cdf7b67-fz8kl |tail -15
  Type    Reason          Age    From               Message
  ----    ------          ----   ----               -------
  Normal  Scheduled       3m51s  default-scheduler  Successfully assigned openshift-compliance/rhcos4-openshift-compliance-pp-5c7cdf7b67-fz8kl to ip-10-0-142-175.us-east-2.compute.internal
  Normal  AddedInterface  3m50s  multus             Add eth0 [10.129.0.42/23] from openshift-sdn
  Normal  Pulling         3m50s  kubelet            Pulling image "quay.io/complianceascode/ocp4:latest"
  Normal  Pulled          3m45s  kubelet            Successfully pulled image "quay.io/complianceascode/ocp4:latest" in 4.729183621s
  Normal  Created         3m44s  kubelet            Created container content-container
  Normal  Started         3m44s  kubelet            Started container content-container
  Normal  Pulling         3m43s  kubelet            Pulling image "quay.io/compliance-operator/compliance-operator:0.1.38"
  Normal  Pulled          3m40s  kubelet            Successfully pulled image "quay.io/compliance-operator/compliance-operator:0.1.38" in 3.479770454s
  Normal  Created         3m39s  kubelet            Created container profileparser
  Normal  Started         3m39s  kubelet            Started container profileparser
  Normal  Pulled          2m36s  kubelet            Container image "quay.io/compliance-operator/compliance-operator:0.1.38" already present on machine
  Normal  Created         2m35s  kubelet            Created container pauser
  Normal  Started         2m35s  kubelet            Started container pauser
Comment 8 Prashant Dhamdhere 2021-08-24 08:30:36 UTC 
[Bug_Verification]

I am able to install Compliance Operator with latest version v0.1.39 using the latest build.
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1706646

$ oc project openshift-compliance
Now using project "openshift-compliance" on server "https://api.pdhamdhe-2348.qe.devcluster.openshift.com:6443".

$ oc get csv
NAME                              DISPLAY                            VERSION    REPLACES   PHASE
compliance-operator.v0.1.39       Compliance Operator                0.1.39                Succeeded
elasticsearch-operator.5.1.1-42   OpenShift Elasticsearch Operator   5.1.1-42              Succeeded

$ oc get sub
NAME                  PACKAGE               SOURCE                CHANNEL
compliance-operator   compliance-operator   compliance-operator   release-0.1

$ oc get pods
NAME                                            READY   STATUS    RESTARTS   AGE
compliance-operator-bb9f644cc-xwfnq             1/1     Running   1          92m
ocp4-openshift-compliance-pp-6d7c7db4bd-jwnnq   1/1     Running   0          91m
rhcos4-openshift-compliance-pp-c7b548bd-9hqvz   1/1     Running   0          91m

$ oc describe pod compliance-operator-bb9f644cc-xwfnq |grep "RELATED_IMAGE"
      RELATED_IMAGE_OPENSCAP:   registry.redhat.io/compliance/openshift-compliance-openscap-rhel8@sha256:1317d69bafd867f358b30168ca40dd49835696821d3f4b1244089f8d46dde3bf
      RELATED_IMAGE_OPERATOR:   registry.redhat.io/compliance/openshift-compliance-rhel8-operator@sha256:10dcd18f4ddf73b295befde8fe720a49d1826d6415b822a4c11a933a8d7a72cb
      RELATED_IMAGE_PROFILE:    registry.redhat.io/compliance/openshift-compliance-content-rhel8@sha256:7527f52d9ce5f22fe5453b23eb490ce1f7e64e0cf7112b13a294c4bc442ae35d
Comment 10 errata-xmlrpc 2021-09-07 06:05:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Compliance Operator bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3214