Bug 1689773 - console operator should set correct message when it's not Available
Summary: console operator should set correct message when it's not Available
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Management Console
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.1.0
Assignee: Jakub Hadvig
QA Contact: Yadan Pei
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-18 04:41 UTC by Yadan Pei
Modified: 2019-06-04 10:46 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Wrong condition message is set on the Available status when console route is not admitted Consequence: User sees wrong message 'Available: No pods available for console deployment.' Fix: Check if route is ready and admitted and set proper message. Result: User sees proper message ("Available: Console route is not admitted") when the Route is not admitted.
Clone Of:
Environment:
Last Closed: 2019-06-04 10:46:01 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:46:06 UTC

Description Yadan Pei 2019-03-18 04:41:50 UTC
Description of problem:
console operator should set correct message when it's not available based on which conditions are not met instead of always setting to "No pods available for console deployment"

Version-Release number of selected component (if applicable):
cluster version: 4.0.0-0.nightly-2019-03-15-063749
console-operator commit: io.openshift.build.commit.url=https://github.com/openshift/console-operator/commit/b9a3350f96176c9b6291c3ba99157237866eb222

How reproducible:
Always

Steps to Reproduce:
1. Change console route to not admitted by router with steps
$ oc edit ingresscontroller -n openshift-ingress-operator
$ oc get ingresscontroller -n openshift-ingress-operator -o yaml
  spec:
    namespaceSelector:
      matchLabels:
        router: prod
$ oc get pods -n openshift-ingress   // ingress pods will be re-created, if new pods are pending, remove old router pods
NAME                              READY   STATUS    RESTARTS   AGE
router-default-8696c5f5bd-qmstz   1/1     Running   0          29m
router-default-8696c5f5bd-xmcjv   1/1     Running   0          25m
$ remove existing console route, set console.operator.spec.managementState to 'Removed' -> then change back to 'Managed', console route will be re-created
$ oc get route -n openshift-console -o yaml  |grep status -A 10
  status:
    ingress: null
kind: List
$ oc get pods -n openshift-console    // console pods are running
NAME                       READY   STATUS    RESTARTS   AGE
console-56c866b54d-6tcx9   1/1     Running   0          21m
console-56c866b54d-vzz7h   1/1     Running   0          21m
$ oc get clusteroperators
NAME                                  VERSION                             AVAILABLE   PROGRESSING   FAILING   SINCE
cluster-autoscaler                    4.0.0-0.nightly-2019-03-15-063749   True        False         False     3h23m
console                               4.0.0-0.nightly-2019-03-15-063749   False       False         False     54s
2. Check console operator status
$ oc get clusteroperators console -o yaml
status:
  conditions:
  - lastTransitionTime: 2019-03-18T01:58:54Z
    reason: AsExpected
    status: "False"
    type: Failing
  - lastTransitionTime: 2019-03-18T01:59:04Z
    reason: AsExpected
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-03-18T01:58:54Z
    message: 'Available: No pods available for console deployment.'
    reason: Available
    status: "False"
    type: Available
  - lastTransitionTime: 2019-03-17T22:41:25Z
    reason: NoData
    status: Unknown
    type: Upgradeable


Actual results:
2. Although console pods are available, but console route is not admitted, console operator is still not Available(Available: False in 'oc get clusteroperators'). However the state message is not correct, it shows "Available: No pods available for console deployment"
1) Since all console pods are available, so it shows 'Available', this is incorrect
https://github.com/openshift/console-operator/blob/cc814fac048a19f34b27d1b8b68b95fe603847e6/pkg/console/operator/status.go#L232
2) we hard coded the message for not Available in https://github.com/openshift/console-operator/blob/cc814fac048a19f34b27d1b8b68b95fe603847e6/pkg/console/operator/status.go#L233

Expected results:
2. A correct message should be something like this "Not Available: console route is not admitted" with correct Available status and reason message.

Additional info:

Comment 1 Jakub Hadvig 2019-03-18 09:45:13 UTC
Fixing PR: https://github.com/openshift/console-operator/pull/178

Comment 2 Yadan Pei 2019-03-21 05:24:25 UTC
Cluster version: 4.0.0-0.nightly-2019-03-20-153904
console-operator commit: 
$ oc get pods -n openshift-console-operator -o yaml|grep image
      image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7a9a9cfac0da2a7f5283d12789b24c64fc89d0e79f21a52fa21f6b049d06d37
$ oc image info quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7a9a9cfac0da2a7f5283d12789b24c64fc89d0e79f21a52fa21f6b049d06d37 | grep commit
             io.openshift.build.commit.id=ffb123d146c32e91f7add5a4cfebd5d9d2e6ad57
             io.openshift.build.commit.url=https://github.com/openshift/console-operator/commit/ffb123d146c32e91f7add5a4cfebd5d9d2e6ad57
$ git log ffb123d146c32e91f7add5a4cfebd5d9d2e6ad57 | grep '#178'
    Merge pull request #178 from jhadvig/BZ1689773


$ following steps above to make console route not admitted
$ oc get route console -n openshift-console -o json | jq '.status.ingress'
null
$ oc get clusteroperator
NAME                                  VERSION                             AVAILABLE   PROGRESSING   FAILING   SINCE
authentication                        4.0.0-0.nightly-2019-03-20-153904   True        False         False     133m
cluster-autoscaler                    4.0.0-0.nightly-2019-03-20-153904   True        False         False     138m
console                               4.0.0-0.nightly-2019-03-20-153904   False       False         False     37s

[yapei@dhcp-140-3 test-files]$ oc get clusteroperator console -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-03-21T01:10:36Z
  generation: 1
  name: console
  resourceVersion: "92634"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/console
  uid: 220929b3-4b76-11e9-b0a1-062547246220
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-03-21T03:23:31Z
    reason: AsExpected
    status: "False"
    type: Failing
  - lastTransitionTime: 2019-03-21T03:23:41Z
    reason: AsExpected
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-03-21T03:23:31Z
    message: 'Available: Console route is not admitted'
    reason: Available
    status: "False"
    type: Available
  - lastTransitionTime: 2019-03-21T01:10:36Z
    reason: NoData
    status: Unknown
    type: Upgradeable
  extension: null
  relatedObjects:
  - group: operator.openshift.io
    name: cluster
    resource: consoles
  - group: config.openshift.io
    name: cluster
    resource: consoles
  - group: oauth.openshift.io
    name: console
    resource: oauthclients
  - group: ""
    name: openshift-console-operator
    resource: namespaces
  - group: ""
    name: openshift-console
    resource: namespaces
  versions:
  - name: operator
    version: 4.0.0-0.nightly-2019-03-20-153904


We can see when route is not Admitted, the reason is 'Available' and message is 'Available: Console route is not admitted', it doesn't look reasonable

Comment 3 Jakub Hadvig 2019-03-25 09:08:30 UTC
The behaviour should be as designed, as I was told by David Eads from Master team.

The message is prefixed because it's a union of many different statuses, so you need to know which it came from.

Here is additional PR for the library-go to show the right reason: https://github.com/openshift/library-go/pull/318

Comment 4 Jakub Hadvig 2019-03-25 13:45:42 UTC
Putting on ON_QA since https://github.com/openshift/library-go/pull/318 got merged.

Regarding the condition Message it should be in the following format:
- "condition.type: condition.msg" -> 'Available: No pods available for console deployment.'

Regarding the condition reason, it's format should be:
- "condition.type+condition.reason" -> 'AvailableRouteNotAdmitted'

Comment 5 Yadan Pei 2019-03-26 03:21:04 UTC
fix PR not merged in 4.0.0-0.nightly-2019-03-25-180911

Comment 7 Yadan Pei 2019-03-29 07:25:41 UTC
https://github.com/openshift/library-go/pull/318 is not merged in 

1. get cluster-kube-api-operator commit in 4.0-art-latest-2019-03-29-040459, it's 325599dcd56e7c040bf070e5179fae0d0729a226
2. Check what changes in library-go has been vendored 
[yapei@dhcp-140-3 cluster-kube-apiserver-operator]$ git log 325599dcd56e7c040bf070e5179fae0d0729a226 -1 -- vendor/github.com/openshift/library-go/
commit b82d6dd22f708a5fbfe2d0dc286ebc3ecbc85a37
Author: Dr. Stefan Schimanski <stefan.schimanski@gmail.com>
Date:   Mon Mar 25 13:20:50 2019 +0100

    bump(*): newest openshift/api

[yapei@dhcp-140-3 cluster-kube-apiserver-operator]$ git log b82d6dd22f708a5fbfe2d0dc286ebc3ecbc85a37 --name-only -1
commit b82d6dd22f708a5fbfe2d0dc286ebc3ecbc85a37
Author: Dr. Stefan Schimanski <stefan.schimanski@gmail.com>
Date:   Mon Mar 25 13:20:50 2019 +0100

    bump(*): newest openshift/api

glide.lock
vendor/github.com/openshift/api/operator/v1/types.go
vendor/github.com/openshift/library-go/alpha-build-machinery/make/targets/openshift/bindata.mk
vendor/github.com/openshift/library-go/alpha-build-machinery/make/targets/openshift/codegen.mk
vendor/github.com/pkg/profile/.travis.yml
vendor/github.com/pkg/profile/profile.go

[yapei@dhcp-140-3 cluster-kube-apiserver-operator]$ git log b82d6dd22f708a5fbfe2d0dc286ebc3ecbc85a37 --patch-with-stat -1 -- glide.lock 
commit b82d6dd22f708a5fbfe2d0dc286ebc3ecbc85a37
Author: Dr. Stefan Schimanski <stefan.schimanski@gmail.com>
Date:   Mon Mar 25 13:20:50 2019 +0100

    bump(*): newest openshift/api
---
 glide.lock | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/glide.lock b/glide.lock
index e80045f1..b4501d95 100644
--- a/glide.lock
+++ b/glide.lock
@@ -1,5 +1,5 @@
 hash: a25f63d9125e16e1bacc281725d5466152d501da1083289aca88b6652d7a400c
-updated: 2019-03-22T11:14:23.669609+01:00
+updated: 2019-03-25T13:20:07.964353+01:00
 imports:
 - name: bitbucket.org/ww/goautoneg
   version: 75cd24fc2f2c2a2088577d12123ddee5f54e0675
@@ -241,7 +241,7 @@ imports:
 - name: github.com/NYTimes/gziphandler
   version: 56545f4a5d46df9a6648819d1664c3a03a13ffdb
 - name: github.com/openshift/api
-  version: 71fdeba186565ceda2c6bde4aac9ea50c42b1c63
+  version: bc6c395d7b3d88cb415dce112eae33908cb72b23
   subpackages:
   - apps
   - apps/v1
@@ -306,7 +306,7 @@ imports:
   - operator/informers/externalversions/operator/v1
   - operator/listers/operator/v1
 - name: github.com/openshift/library-go                          //library-go changes
-  version: c808b4f42728549453f702c5bdc21413f02f09d4
+  version: 035149b6b1185de7572de96df2dc51570fcf85c7
   subpackages:
   - cmd/crd-schema-gen/generator
   - pkg/assets
....
[yapei@dhcp-140-3 library-go]$ git log 035149b6b1185de7572de96df2dc51570fcf85c7 | grep '#318'    //nothing returns, pr 318 is not included yet in 4.0-art-latest-2019-03-29-040459

Comment 8 Yadan Pei 2019-04-04 08:14:53 UTC
Following same steps to check if fix PR is merged in 4.0.0-0.nightly-2019-04-02-150843

[yapei@dhcp-140-3 library-go]$ git log ce0e892919bd98ca70214f34ba4d59428f76bc3c | grep '#318'
    Merge pull request #318 from jhadvig/union-reason            // PR is merged in 4.0.0-0.nightly-2019-04-02-150843



And change route not admitted

$ oc get route console -o yaml -n openshift-console | grep -i status -A 10
status:
  ingress: null
$ oc get clusteroperators
NAME                                 VERSION                             AVAILABLE   PROGRESSING   FAILING   SINCE
authentication                       4.0.0-0.nightly-2019-04-02-150843   False       True          False     3m37s
cloud-credential                     4.0.0-0.nightly-2019-04-02-150843   True        False         False     33h
cluster-autoscaler                   4.0.0-0.nightly-2019-04-02-150843   True        False         False     33h
console                              4.0.0-0.nightly-2019-04-02-150843   False       False         False     106s
$ oc get clusteroperator console -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
.....
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-04-02T22:57:33Z
    reason: AsExpected
    status: "False"
    type: Failing
  - lastTransitionTime: 2019-04-04T08:00:38Z
    reason: AsExpected
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-04-04T08:00:22Z
    message: 'Available: Console route is not admitted'
    reason: Available
    status: "False"
    type: Available


it shows "Console route is not admitted", move to VERIFIED

Comment 10 errata-xmlrpc 2019-06-04 10:46:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.