Bug 1962310

Summary: operator-lifecycle-manager-packageserver flaps Available=False despite unavailableReplicas <= maxUnavailable
Product: OpenShift Container Platform Reporter: Ben Luddy <bluddy>
Component: OLMAssignee: Ben Luddy <bluddy>
OLM sub component: OLM QA Contact: Bruno Andrade <bandrade>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: bandrade, bluddy, davegord, jiazha, nhale, scolange, wking
Version: 4.8Keywords: Triaged, Upgrades
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1959009 Environment:
[bz-OLM] clusteroperator/operator-lifecycle-manager-packageserver should not change condition/Available
Last Closed: 2021-06-08 13:54:23 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1959009    
Bug Blocks:    

Comment 5 Salvatore Colangelo 2021-06-01 20:22:45 UTC
[scolange@scolange extended]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.17    True        False         4h24m   Cluster version is 4.6.17



spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: packageserver
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        olm.operatorGroup: olm-operators
        olm.operatorNamespace: openshift-operator-lifecycle-manager
        olm.targetNamespaces: openshift-operator-lifecycle-manager
        olmcahash: ab1f18b6b07ea703d75b1c8d3da7aa5c5b38216bbafd9a4cb3a75fb168e37f1d
      creationTimestamp: null
      labels:
--

Not Verfied!

Comment 6 W. Trevor King 2021-06-01 20:48:44 UTC
Why the 4.6.17 test above?  That's a pretty old version.  From [1], looks like this is just in nightlies since 4.6.31.  Moving back to ON_QA, because we want to look at this before those landed 4.6 nightlies become a new 4.6.z release.

[1]: https://amd64.ocp.releases.ci.openshift.org/releasestream/4.6.0-0.nightly/release/4.6.0-0.nightly-2021-05-31-074224?from=4.6.31

Comment 7 Bruno Andrade 2021-06-02 17:14:20 UTC
OLM version: 0.16.1
git commit: bf72c5d00150acd951894f256cf2e65749326213
4.6.0-0.nightly-2021-05-31-074224

 oc get deployment packageserver -n openshift-operator-lifecycle-manager -o yaml | grep "spec:" -A 20
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: packageserver
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        olm.operatorGroup: olm-operators
        olm.operatorNamespace: openshift-operator-lifecycle-manager
        olm.targetNamespaces: openshift-operator-lifecycle-manager
        olmcahash: 8489e8e82a0f34560df3f0dddfac59cfe6eda65313b6fba3c46ea3ad9d7ad642
      creationTimestamp: null
      labels:

Don't see any Available=False issues in CI: realted with 4.6

 w3m -dump -cols 200 'https://search.ci.openshift.org/?search=clusteroperator%2Foperator-lifecycle-manager-packageserver+condition%2FAvailable+status%2FFalse&maxAge=24h&type=junit' | grep 'failures match' | grep -v 'rehearse-\|pull-ci-'
periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade (all) - 17 runs, 94% failed, 100% of failures match = 94% impact
periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-vsphere-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-ovn-upgrade (all) - 18 runs, 100% failed, 89% of failures match = 89% impact
periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-ovirt-upgrade (all) - 2 runs, 100% failed, 100% of failures match = 100% impact
periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-gcp-ovn-upgrade (all) - 4 runs, 100% failed, 100% of failures match = 100% impact
periodic-ci-openshift-release-master-ci-4.8-e2e-gcp-upgrade (all) - 17 runs, 29% failed, 20% of failures match = 6% impact
periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-azure-ovn-upgrade (all) - 4 runs, 100% failed, 25% of failures match = 25% impact
release-openshift-ocp-installer-upgrade-remote-libvirt-s390x-4.7-to-4.8 (all) - 2 runs, 100% failed, 100% of failures match = 100% impact
periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-azure-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-gcp-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
periodic-ci-openshift-release-master-nightly-4.8-upgrade-from-stable-4.7-e2e-metal-ipi-upgrade (all) - 5 runs, 100% failed, 80% of failures match = 80% impact
periodic-ci-openshift-release-master-nightly-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade (all) - 5 runs, 100% failed, 100% of failures match = 100% impact
periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-compact-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-upgrade (all) - 5 runs, 60% failed, 33% of failures match = 20% impact
release-openshift-origin-installer-launch-gcp (all) - 124 runs, 33% failed, 2% of failures match = 1% impact
periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-from-stable-4.7-e2e-aws-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
periodic-ci-openshift-release-master-ci-4.9-e2e-aws-upgrade-single-node (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-e2e-aws-compact-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-ovn-upgrade-rollback (all) - 1 runs, 100% failed, 100% of failures match = 100% impact

Marking as VERIFIED

Comment 9 errata-xmlrpc 2021-06-08 13:54:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.32 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2157