Hide Forgot
Description of problem: console operator should set correct message when it's not available based on which conditions are not met instead of always setting to "No pods available for console deployment" Version-Release number of selected component (if applicable): cluster version: 4.0.0-0.nightly-2019-03-15-063749 console-operator commit: io.openshift.build.commit.url=https://github.com/openshift/console-operator/commit/b9a3350f96176c9b6291c3ba99157237866eb222 How reproducible: Always Steps to Reproduce: 1. Change console route to not admitted by router with steps $ oc edit ingresscontroller -n openshift-ingress-operator $ oc get ingresscontroller -n openshift-ingress-operator -o yaml spec: namespaceSelector: matchLabels: router: prod $ oc get pods -n openshift-ingress // ingress pods will be re-created, if new pods are pending, remove old router pods NAME READY STATUS RESTARTS AGE router-default-8696c5f5bd-qmstz 1/1 Running 0 29m router-default-8696c5f5bd-xmcjv 1/1 Running 0 25m $ remove existing console route, set console.operator.spec.managementState to 'Removed' -> then change back to 'Managed', console route will be re-created $ oc get route -n openshift-console -o yaml |grep status -A 10 status: ingress: null kind: List $ oc get pods -n openshift-console // console pods are running NAME READY STATUS RESTARTS AGE console-56c866b54d-6tcx9 1/1 Running 0 21m console-56c866b54d-vzz7h 1/1 Running 0 21m $ oc get clusteroperators NAME VERSION AVAILABLE PROGRESSING FAILING SINCE cluster-autoscaler 4.0.0-0.nightly-2019-03-15-063749 True False False 3h23m console 4.0.0-0.nightly-2019-03-15-063749 False False False 54s 2. Check console operator status $ oc get clusteroperators console -o yaml status: conditions: - lastTransitionTime: 2019-03-18T01:58:54Z reason: AsExpected status: "False" type: Failing - lastTransitionTime: 2019-03-18T01:59:04Z reason: AsExpected status: "False" type: Progressing - lastTransitionTime: 2019-03-18T01:58:54Z message: 'Available: No pods available for console deployment.' reason: Available status: "False" type: Available - lastTransitionTime: 2019-03-17T22:41:25Z reason: NoData status: Unknown type: Upgradeable Actual results: 2. Although console pods are available, but console route is not admitted, console operator is still not Available(Available: False in 'oc get clusteroperators'). However the state message is not correct, it shows "Available: No pods available for console deployment" 1) Since all console pods are available, so it shows 'Available', this is incorrect https://github.com/openshift/console-operator/blob/cc814fac048a19f34b27d1b8b68b95fe603847e6/pkg/console/operator/status.go#L232 2) we hard coded the message for not Available in https://github.com/openshift/console-operator/blob/cc814fac048a19f34b27d1b8b68b95fe603847e6/pkg/console/operator/status.go#L233 Expected results: 2. A correct message should be something like this "Not Available: console route is not admitted" with correct Available status and reason message. Additional info:
Fixing PR: https://github.com/openshift/console-operator/pull/178
Cluster version: 4.0.0-0.nightly-2019-03-20-153904 console-operator commit: $ oc get pods -n openshift-console-operator -o yaml|grep image image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7a9a9cfac0da2a7f5283d12789b24c64fc89d0e79f21a52fa21f6b049d06d37 $ oc image info quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7a9a9cfac0da2a7f5283d12789b24c64fc89d0e79f21a52fa21f6b049d06d37 | grep commit io.openshift.build.commit.id=ffb123d146c32e91f7add5a4cfebd5d9d2e6ad57 io.openshift.build.commit.url=https://github.com/openshift/console-operator/commit/ffb123d146c32e91f7add5a4cfebd5d9d2e6ad57 $ git log ffb123d146c32e91f7add5a4cfebd5d9d2e6ad57 | grep '#178' Merge pull request #178 from jhadvig/BZ1689773 $ following steps above to make console route not admitted $ oc get route console -n openshift-console -o json | jq '.status.ingress' null $ oc get clusteroperator NAME VERSION AVAILABLE PROGRESSING FAILING SINCE authentication 4.0.0-0.nightly-2019-03-20-153904 True False False 133m cluster-autoscaler 4.0.0-0.nightly-2019-03-20-153904 True False False 138m console 4.0.0-0.nightly-2019-03-20-153904 False False False 37s [yapei@dhcp-140-3 test-files]$ oc get clusteroperator console -o yaml apiVersion: config.openshift.io/v1 kind: ClusterOperator metadata: creationTimestamp: 2019-03-21T01:10:36Z generation: 1 name: console resourceVersion: "92634" selfLink: /apis/config.openshift.io/v1/clusteroperators/console uid: 220929b3-4b76-11e9-b0a1-062547246220 spec: {} status: conditions: - lastTransitionTime: 2019-03-21T03:23:31Z reason: AsExpected status: "False" type: Failing - lastTransitionTime: 2019-03-21T03:23:41Z reason: AsExpected status: "False" type: Progressing - lastTransitionTime: 2019-03-21T03:23:31Z message: 'Available: Console route is not admitted' reason: Available status: "False" type: Available - lastTransitionTime: 2019-03-21T01:10:36Z reason: NoData status: Unknown type: Upgradeable extension: null relatedObjects: - group: operator.openshift.io name: cluster resource: consoles - group: config.openshift.io name: cluster resource: consoles - group: oauth.openshift.io name: console resource: oauthclients - group: "" name: openshift-console-operator resource: namespaces - group: "" name: openshift-console resource: namespaces versions: - name: operator version: 4.0.0-0.nightly-2019-03-20-153904 We can see when route is not Admitted, the reason is 'Available' and message is 'Available: Console route is not admitted', it doesn't look reasonable
The behaviour should be as designed, as I was told by David Eads from Master team. The message is prefixed because it's a union of many different statuses, so you need to know which it came from. Here is additional PR for the library-go to show the right reason: https://github.com/openshift/library-go/pull/318
Putting on ON_QA since https://github.com/openshift/library-go/pull/318 got merged. Regarding the condition Message it should be in the following format: - "condition.type: condition.msg" -> 'Available: No pods available for console deployment.' Regarding the condition reason, it's format should be: - "condition.type+condition.reason" -> 'AvailableRouteNotAdmitted'
fix PR not merged in 4.0.0-0.nightly-2019-03-25-180911
https://github.com/openshift/library-go/pull/318 is not merged in 1. get cluster-kube-api-operator commit in 4.0-art-latest-2019-03-29-040459, it's 325599dcd56e7c040bf070e5179fae0d0729a226 2. Check what changes in library-go has been vendored [yapei@dhcp-140-3 cluster-kube-apiserver-operator]$ git log 325599dcd56e7c040bf070e5179fae0d0729a226 -1 -- vendor/github.com/openshift/library-go/ commit b82d6dd22f708a5fbfe2d0dc286ebc3ecbc85a37 Author: Dr. Stefan Schimanski <stefan.schimanski> Date: Mon Mar 25 13:20:50 2019 +0100 bump(*): newest openshift/api [yapei@dhcp-140-3 cluster-kube-apiserver-operator]$ git log b82d6dd22f708a5fbfe2d0dc286ebc3ecbc85a37 --name-only -1 commit b82d6dd22f708a5fbfe2d0dc286ebc3ecbc85a37 Author: Dr. Stefan Schimanski <stefan.schimanski> Date: Mon Mar 25 13:20:50 2019 +0100 bump(*): newest openshift/api glide.lock vendor/github.com/openshift/api/operator/v1/types.go vendor/github.com/openshift/library-go/alpha-build-machinery/make/targets/openshift/bindata.mk vendor/github.com/openshift/library-go/alpha-build-machinery/make/targets/openshift/codegen.mk vendor/github.com/pkg/profile/.travis.yml vendor/github.com/pkg/profile/profile.go [yapei@dhcp-140-3 cluster-kube-apiserver-operator]$ git log b82d6dd22f708a5fbfe2d0dc286ebc3ecbc85a37 --patch-with-stat -1 -- glide.lock commit b82d6dd22f708a5fbfe2d0dc286ebc3ecbc85a37 Author: Dr. Stefan Schimanski <stefan.schimanski> Date: Mon Mar 25 13:20:50 2019 +0100 bump(*): newest openshift/api --- glide.lock | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/glide.lock b/glide.lock index e80045f1..b4501d95 100644 --- a/glide.lock +++ b/glide.lock @@ -1,5 +1,5 @@ hash: a25f63d9125e16e1bacc281725d5466152d501da1083289aca88b6652d7a400c -updated: 2019-03-22T11:14:23.669609+01:00 +updated: 2019-03-25T13:20:07.964353+01:00 imports: - name: bitbucket.org/ww/goautoneg version: 75cd24fc2f2c2a2088577d12123ddee5f54e0675 @@ -241,7 +241,7 @@ imports: - name: github.com/NYTimes/gziphandler version: 56545f4a5d46df9a6648819d1664c3a03a13ffdb - name: github.com/openshift/api - version: 71fdeba186565ceda2c6bde4aac9ea50c42b1c63 + version: bc6c395d7b3d88cb415dce112eae33908cb72b23 subpackages: - apps - apps/v1 @@ -306,7 +306,7 @@ imports: - operator/informers/externalversions/operator/v1 - operator/listers/operator/v1 - name: github.com/openshift/library-go //library-go changes - version: c808b4f42728549453f702c5bdc21413f02f09d4 + version: 035149b6b1185de7572de96df2dc51570fcf85c7 subpackages: - cmd/crd-schema-gen/generator - pkg/assets .... [yapei@dhcp-140-3 library-go]$ git log 035149b6b1185de7572de96df2dc51570fcf85c7 | grep '#318' //nothing returns, pr 318 is not included yet in 4.0-art-latest-2019-03-29-040459
Following same steps to check if fix PR is merged in 4.0.0-0.nightly-2019-04-02-150843 [yapei@dhcp-140-3 library-go]$ git log ce0e892919bd98ca70214f34ba4d59428f76bc3c | grep '#318' Merge pull request #318 from jhadvig/union-reason // PR is merged in 4.0.0-0.nightly-2019-04-02-150843 And change route not admitted $ oc get route console -o yaml -n openshift-console | grep -i status -A 10 status: ingress: null $ oc get clusteroperators NAME VERSION AVAILABLE PROGRESSING FAILING SINCE authentication 4.0.0-0.nightly-2019-04-02-150843 False True False 3m37s cloud-credential 4.0.0-0.nightly-2019-04-02-150843 True False False 33h cluster-autoscaler 4.0.0-0.nightly-2019-04-02-150843 True False False 33h console 4.0.0-0.nightly-2019-04-02-150843 False False False 106s $ oc get clusteroperator console -o yaml apiVersion: config.openshift.io/v1 kind: ClusterOperator metadata: ..... spec: {} status: conditions: - lastTransitionTime: 2019-04-02T22:57:33Z reason: AsExpected status: "False" type: Failing - lastTransitionTime: 2019-04-04T08:00:38Z reason: AsExpected status: "False" type: Progressing - lastTransitionTime: 2019-04-04T08:00:22Z message: 'Available: Console route is not admitted' reason: Available status: "False" type: Available it shows "Console route is not admitted", move to VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758