Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1715612

Summary:

Upgrade claimed to be successful when cloud-credential-operator could not be upgraded due to crashloop

Product:

OpenShift Container Platform

Reporter:

Mike Fiedler <mifiedle>

Component:

Cloud Credential Operator

Assignee:

Devan Goodwin <dgoodwin>

Status:

CLOSED DUPLICATE

QA Contact:

Oleg Nesterov <olnester>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

4.1.0

CC:

aos-bugs, bleanhar, esimard, jokerman, mmccomas, wking

Target Milestone:

---

Target Release:

4.1.z

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2019-08-02 17:14:28 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
CVO and CCO artifacts	none

Description Mike Fiedler 2019-05-30 19:13:13 UTC

Description of problem:

With cloud-credential-operator crashlooping (see bug 1711402) I performed an upgrade from 4.1.0-0.nightly-2019-05-23-213243 to 4.1.0-0.nightly-2019-05-24-040103.   The upgrade claimed to be successful even though the cloud-credential-operator never upgraded.

I think this blocks fixing via upgrade an operator in this type of condition 


NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-05-24-040103   True        False         17m     Cluster version is 4.1.0-0.nightly-2019-05-24-040103

NAME                                 VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                       4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
cloud-credential                     4.1.0-0.nightly-2019-05-23-213243   True        False         False      22h
cluster-autoscaler                   4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
console                              4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
dns                                  4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
image-registry                       4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
ingress                              4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
kube-apiserver                       4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
kube-controller-manager              4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
kube-scheduler                       4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
machine-api                          4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
machine-config                       4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
marketplace                          4.1.0-0.nightly-2019-05-24-040103   True        False         False      21m
monitoring                           4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
network                              4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
node-tuning                          4.1.0-0.nightly-2019-05-24-040103   True        False         False      22m
openshift-apiserver                  4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
openshift-controller-manager         4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
openshift-samples                    4.1.0-0.nightly-2019-05-24-040103   True        False         False      22m
operator-lifecycle-manager           4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
operator-lifecycle-manager-catalog   4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
service-ca                           4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
service-catalog-apiserver            4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
service-catalog-controller-manager   4.1.0-0.nightly-2019-05-24-040103   True        False         False      22h
storage                              4.1.0-0.nightly-2019-05-24-040103   True        False         False      22m

Reproducer:

1. Create about 2000 projects each with 20 secrets until the cloud-credential-operator exceeds its 500Mi memory limit and starts to crashloop
2. Upgrade the cluster

Comment 4 Mike Fiedler 2019-05-31 12:11:26 UTC

Created attachment 1575679 [details]
CVO and CCO artifacts

Attaching additional requested info.  Hopefully the CVO and CCO pod logs haven't wrapped.   We'll reproduce fresh today in any case.

Comment 5 Abhinav Dahiya 2019-06-24 20:35:36 UTC

Is this still reproducible. This is not a bug with CVO as CCO is reporting the expected version falsely and there isn't much CVO can do.

Comment 6 W. Trevor King 2019-08-01 23:49:29 UTC

Close as a dup of 1727943 (or vice versa)?

Comment 7 Devan Goodwin 2019-08-02 17:14:28 UTC


*** This bug has been marked as a duplicate of bug 1727943 ***