Bug 1995190

Summary: Community operator etcd fails to install on 4.9
Product: OpenShift Container Platform Reporter: Tom Buskey <tbuskey>
Component: ISV OperatorsAssignee: tonyc
Status: CLOSED ERRATA QA Contact: Tom Buskey <tbuskey>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.9CC: aos-bugs, bandrade, jiazha, jitli, kuiwang, scolange, sd-ecosystem, tbuskey, xzha
Target Milestone: ---   
Target Release: 4.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-10 21:01:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tom Buskey 2021-08-18 15:38:36 UTC
Description of problem:
Installling etcd from operator hub stays at Pending

Version-Release number of selected component (if applicable):
oc version
Client Version: 4.9.0-202108170012.p0.git.681248e.assembly.stream-681248e
Server Version: 4.9.0-0.nightly-2021-08-18-033031
Kubernetes Version: v1.22.0-rc.0+f967870

etcd 0.9.4

How reproducible:

always
Steps to Reproduce:
1. Install etcd from operator hub into a single namespace
2.
3.

Actual results:
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.4   etcd      0.9.4     etcdoperator.v0.9.2   Pending

oc get sub etcd -o yaml
...
conditions:
  - lastTransitionTime: "2021-08-18T15:22:02Z"
    message: all available catalogsources are healthy
    reason: AllCatalogSourcesHealthy
    status: "False"
    type: CatalogSourcesUnhealthy
  - lastTransitionTime: "2021-08-18T15:23:04Z"
    message: 'api-server resource not found installing CustomResourceDefinition etcdrestores.etcd.database.coreos.com:
      GroupVersionKind apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition
      not found on the cluster. This API may have been deprecated and removed, see
      https://kubernetes.io/docs/reference/using-api/deprecation-guide/ for more information.'
    reason: InstallComponentFailed


Expected results:
etcd should succeed

Additional info:

Comment 2 Jian Zhang 2021-09-08 08:04:03 UTC
Failed to install the etcd-operator that from the Community CatalogSource. Get the below errors:
2024 time="2021-09-08T07:51:02Z" level=info msg="Bundle unpacking failed. Reason: DeadlineExceeded, and Message: Job was active longer than spe     cified deadline" id=Uij6V ip=install-6rbg4 namespace=default phase=Installing

The detailed info as follows:
[cloud-user@preserve-olm-env etcd]$ oc get sub etcd -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  creationTimestamp: "2021-09-08T07:50:52Z"
  generation: 1
  labels:
    operators.coreos.com/etcd.default: ""
  name: etcd
  namespace: default
  resourceVersion: "83451"
  uid: 5304da79-33b3-4c90-91c2-70ba06929978
spec:
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4
status:
  catalogHealth:
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: certified-operators
      namespace: openshift-marketplace
      resourceVersion: "79606"
      uid: 0026efcd-69f2-4c08-a14f-62ec9cb4a9c0
    healthy: true
    lastUpdated: "2021-09-08T07:51:02Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: community-operators
      namespace: openshift-marketplace
      resourceVersion: "81722"
      uid: baaacb0e-4f26-4488-b49a-44876c75da00
    healthy: true
    lastUpdated: "2021-09-08T07:51:02Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: redhat-marketplace
      namespace: openshift-marketplace
      resourceVersion: "81251"
      uid: 38742052-fb79-4574-aea3-00d6c5164ad5
    healthy: true
    lastUpdated: "2021-09-08T07:51:02Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: redhat-operators
      namespace: openshift-marketplace
      resourceVersion: "82741"
      uid: 42e79b16-6283-4626-97ad-e7c0f938a2f8
    healthy: true
    lastUpdated: "2021-09-08T07:51:02Z"
  conditions:
  - lastTransitionTime: "2021-09-08T07:51:02Z"
    message: all available catalogsources are healthy
    reason: AllCatalogSourcesHealthy
    status: "False"
    type: CatalogSourcesUnhealthy
  - status: "False"
    type: ResolutionFailed
  - lastTransitionTime: "2021-09-08T07:51:03Z"
    message: 'Bundle unpacking failed. Reason: DeadlineExceeded, and Message: Job
      was active longer than specified deadline'
    reason: InstallCheckFailed
    status: "True"
    type: InstallPlanFailed
  currentCSV: etcdoperator.v0.9.4
  installPlanGeneration: 1
  installPlanRef:
    apiVersion: operators.coreos.com/v1alpha1
    kind: InstallPlan
    name: install-6rbg4
    namespace: default
    resourceVersion: "83429"
    uid: ac812d46-9a91-4c88-91ed-798414670257
  installplan:
    apiVersion: operators.coreos.com/v1alpha1
    kind: InstallPlan
    name: install-6rbg4
    uuid: ac812d46-9a91-4c88-91ed-798414670257
  lastUpdated: "2021-09-08T07:51:03Z"
  state: UpgradePending


[cloud-user@preserve-olm-env etcd]$ oc get ip
NAME            CSV                   APPROVAL    APPROVED
install-6rbg4   etcdoperator.v0.9.4   Automatic   true
[cloud-user@preserve-olm-env etcd]$ oc get csv
No resources found in default namespace.

[cloud-user@preserve-olm-env etcd]$ oc get ip install-6rbg4 -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: InstallPlan
metadata:
  creationTimestamp: "2021-09-08T07:51:02Z"
  generateName: install-
  generation: 1
  labels:
    operators.coreos.com/etcd.default: ""
  name: install-6rbg4
  namespace: default
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: Subscription
    name: etcd
    uid: 5304da79-33b3-4c90-91c2-70ba06929978
  resourceVersion: "83441"
  uid: ac812d46-9a91-4c88-91ed-798414670257
spec:
  approval: Automatic
  approved: true
  clusterServiceVersionNames:
  - etcdoperator.v0.9.4
  generation: 1
status:
  bundleLookups:
  - catalogSourceRef:
      name: community-operators
      namespace: openshift-marketplace
    conditions:
    - message: bundle contents have not yet been persisted to installplan status
      reason: BundleNotUnpacked
      status: "True"
      type: BundleLookupNotPersisted
    - message: unpack job not yet started
      reason: JobNotStarted
      status: "True"
      type: BundleLookupPending
    - lastTransitionTime: "2021-09-08T07:51:02Z"
      message: Job was active longer than specified deadline
      reason: DeadlineExceeded
      status: "True"
      type: BundleLookupFailed
    identifier: etcdoperator.v0.9.4
    path: quay.io/openshift-community-operators/etcd@sha256:6334a904a229eb1d51a6708dd774e23bf6d04ab20cc5e9efffa363791dce90a8
    properties: '{"properties":[{"type":"olm.gvk","value":{"group":"etcd.database.coreos.com","kind":"EtcdBackup","version":"v1beta2"}},{"type":"olm.gvk","value":{"group":"etcd.database.coreos.com","kind":"EtcdCluster","version":"v1beta2"}},{"type":"olm.gvk","value":{"group":"etcd.database.coreos.com","kind":"EtcdRestore","version":"v1beta2"}},{"type":"olm.maxOpenShiftVersion","value":"4.8"},{"type":"olm.package","value":{"packageName":"etcd","version":"0.9.4"}}]}'
    replaces: etcdoperator.v0.9.2
  catalogSources: []
  conditions:
  - lastTransitionTime: "2021-09-08T07:51:02Z"
    lastUpdateTime: "2021-09-08T07:51:02Z"
    message: 'Bundle unpacking failed. Reason: DeadlineExceeded, and Message: Job
      was active longer than specified deadline'
    reason: InstallCheckFailed
    status: "False"
    type: Installed
  phase: Failed

[cloud-user@preserve-olm-env etcd]$ oc get catalogsource -n openshift-marketplace community-operators -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  annotations:
    operatorframework.io/managed-by: marketplace-operator
    operatorframework.io/priorityclass: system-cluster-critical
    target.workload.openshift.io/management: '{"effect": "PreferredDuringScheduling"}'
  creationTimestamp: "2021-09-08T04:59:55Z"
  generation: 1
  name: community-operators
  namespace: openshift-marketplace
  resourceVersion: "85128"
  uid: baaacb0e-4f26-4488-b49a-44876c75da00
spec:
  displayName: Community Operators
  icon:
    base64data: ""
    mediatype: ""
  image: registry.redhat.io/redhat/community-operator-index:v4.9
  priority: -400
  publisher: Red Hat
  sourceType: grpc
  updateStrategy:
    registryPoll:
      interval: 10m0s
status:
  connectionState:
    address: community-operators.openshift-marketplace.svc:50051
    lastConnect: "2021-09-08T06:56:05Z"
    lastObservedState: READY
  latestImageRegistryPoll: "2021-09-08T07:55:59Z"
  registryService:
    createdAt: "2021-09-08T04:59:55Z"
    port: "50051"
    protocol: grpc
    serviceName: community-operators
    serviceNamespace: openshift-marketplace


[cloud-user@preserve-olm-env etcd]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-09-07-201519   True        False         159m    Cluster version is 4.9.0-0.nightly-2021-09-07-201519

Comment 3 Tom Buskey 2021-10-28 16:39:23 UTC
Verifying

oc version
Client Version: 4.9.0-202110262214.p0.git.96e95ce.assembly.stream-96e95ce
Server Version: 4.9.0-0.nightly-2021-10-28-052726
Kubernetes Version: v1.22.0-rc.0+a44d0f0

Installed successfully from OperatorHub

Comment 6 errata-xmlrpc 2021-11-10 21:01:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.6 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4119