Created attachment 1591904 [details] Log from cluster-version-operator Description of problem: dns, network and machine-config cannot be upgraded from 4.1.6 to 4.2 nightly: $ oc get clusteroperator NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.2.0-0.nightly-2019-07-18-032637 True False False 21h cloud-credential 4.2.0-0.nightly-2019-07-18-032637 True False False 21h cluster-autoscaler 4.2.0-0.nightly-2019-07-18-032637 True False False 21h console 4.2.0-0.nightly-2019-07-18-032637 True False False 21h dns 4.1.6 True False False 21h image-registry 4.2.0-0.nightly-2019-07-18-032637 True False False 20h ingress 4.2.0-0.nightly-2019-07-15-231921 True False False 21h kube-apiserver 4.2.0-0.nightly-2019-07-18-032637 True False False 21h kube-controller-manager 4.2.0-0.nightly-2019-07-18-032637 True False False 21h kube-scheduler 4.2.0-0.nightly-2019-07-18-032637 True False False 21h machine-api 4.2.0-0.nightly-2019-07-18-032637 True False False 21h machine-config 4.1.6 True False False 21h marketplace 4.2.0-0.nightly-2019-07-18-032637 True False False 19h monitoring 4.2.0-0.nightly-2019-07-18-032637 True False False 19h network 4.1.6 True False False 21h node-tuning 4.2.0-0.nightly-2019-07-18-032637 True False False 19h openshift-apiserver 4.2.0-0.nightly-2019-07-18-032637 True False False 19h openshift-controller-manager 4.2.0-0.nightly-2019-07-18-032637 True False False 21h openshift-samples 4.2.0-0.nightly-2019-07-18-032637 True False False 19h operator-lifecycle-manager 4.2.0-0.nightly-2019-07-18-032637 True False False 21h operator-lifecycle-manager-catalog 4.2.0-0.nightly-2019-07-18-032637 True False False 21h operator-lifecycle-manager-packageserver False False False 19h service-ca 4.2.0-0.nightly-2019-07-18-032637 True False False 21h service-catalog-apiserver 4.2.0-0.nightly-2019-07-18-032637 True False False 21h service-catalog-controller-manager 4.2.0-0.nightly-2019-07-18-032637 True False False 21h storage 4.2.0-0.nightly-2019-07-18-032637 True False False 19h support 4.2.0-0.nightly-2019-07-18-032637 True False False 19h Version-Release number of selected component (if applicable): 4.1.4->4.1.6->4.2 nightly build How reproducible: Always Steps to Reproduce: 1.Prepare a 4.1 cluster 2.oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.2.0-0.nightly-2019-07-18-032637 --force=true 3. Actual results: dns, network and machine-config remains 4.1 version Expected results: They should be upgrade like other Operators. Additional info: CVO log is in attachment.
as apparent from the logs, CVO is waiting for ingress-operator deployment to roll-out. it cannot proceed without that. can you include `oc adm must-gather` to debug why?
CVO logs shows: oc logs -f console-operator-756fbfcf59-dq7z7 -n openshift-console-operator .... W0729 08:07:19.790348 1 reflector.go:289] github.com/openshift/client-go/route/informers/externalversions/factory.go:101: watch of *v1.Route ended with: The resourceVersion for the provided watch is too old. W0729 08:08:20.738596 1 reflector.go:289] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: watch of *v1.OAuthClient ended with: The resourceVersion for the provided watch is too old. ....
Seems the root cause is ingress operator and have opened two separate BUGs to track them: [1] https://bugzilla.redhat.com/show_bug.cgi?id=1734192 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1734414 Since the first one has been verified on AWS platform so I'm closing this one.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922