Bug 1840616
| Summary: | Upgrading a cluster from 4.3.18 to 4.4.4 did not works when using a custom policy for kube-scheduler | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Andreas Nowak <anowak> |
| Component: | kube-scheduler | Assignee: | Jan Chaloupka <jchaloup> |
| Status: | CLOSED ERRATA | QA Contact: | RamaKasturi <knarra> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.4 | CC: | aos-bugs, jchaloup, ksalunkh, mfojtik |
| Target Milestone: | --- | ||
| Target Release: | 4.6.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-10-27 16:01:07 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Comment 14
RamaKasturi
2020-06-24 06:05:51 UTC
Updated Docs PR: https://github.com/openshift/openshift-docs/pull/23346 > @jan, one question if this a documentation related bug, why do not we change the component as well to the documentation ? Changing the component will make harder to find the issue again. Tried to verify the bug by upgrading from 4.3.18 -> 4.4.9 -> 4.5 nightly -> 4.6. Was able to successfully upgrade by having the mentioned settings in the policy.cfg file and did not hit any issue. No issues reported from kube-scheduler as well.
[ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.3.18 True False 82m Cluster version is 4.3.18
[ramakasturinarra@dhcp35-60 ~]$ oc logs -f openshift-kube-scheduler-ip-10-0-132-202.us-east-2.compute.internal -n openshift-kube-scheduler | grep predicates
I0727 07:13:27.918092 1 factory.go:382] Creating scheduler with fit predicates 'map[CheckNodeUnschedulable:{} CheckVolumeBinding:{} GeneralPredicates:{} MatchInterPodAffinity:{} MaxAzureDiskVolumeCount:{} MaxCSIVolumeCountPred:{} MaxEBSVolumeCount:{} MaxGCEPDVolumeCount:{} NoDiskConflict:{} NoVolumeZoneConflict:{} PodToleratesNodeTaints:{}]' and priority functions 'map[BalancedResourceAllocation:{} EqualPriority:{} ImageLocalityPriority:{} InterPodAffinityPriority:{} LeastRequestedPriority:{} NodeAffinityPriority:{} NodePreferAvoidPodsPriority:{} SelectorSpreadPriority:{} ServiceSpreadingPriority:{} TaintTolerationPriority:{}]'
[ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.4.9 True False 17m Cluster version is 4.4.9
[ramakasturinarra@dhcp35-60 ~]$ oc get co kube-scheduler
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
kube-scheduler 4.4.9 True False False 57m
[ramakasturinarra@dhcp35-60 ~]$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.4.9 True False False 150m
cloud-credential 4.4.9 True False False 167m
cluster-autoscaler 4.4.9 True False False 157m
console 4.4.9 True False False 24m
csi-snapshot-controller 4.4.9 True False False 25m
dns 4.4.9 True False False 162m
etcd 4.4.9 True False False 60m
image-registry 4.4.9 True False False 27m
ingress 4.4.9 True False False 26m
insights 4.4.9 True False False 158m
kube-apiserver 4.4.9 True False False 161m
kube-controller-manager 4.4.9 True False False 57m
kube-scheduler 4.4.9 True False False 57m
kube-storage-version-migrator 4.4.9 True False False 27m
machine-api 4.4.9 True False False 162m
machine-config 4.4.9 True False False 162m
marketplace 4.4.9 True False False 28m
monitoring 4.4.9 True False False 58m
network 4.4.9 True False False 163m
node-tuning 4.4.9 True False False 7m59s
openshift-apiserver 4.4.9 True False False 34m
openshift-controller-manager 4.4.9 True False False 161m
openshift-samples 4.4.9 True False False 51m
operator-lifecycle-manager 4.4.9 True False False 158m
operator-lifecycle-manager-catalog 4.4.9 True False False 158m
operator-lifecycle-manager-packageserver 4.4.9 True False False 24m
service-ca 4.4.9 True False False 163m
service-catalog-apiserver 4.4.9 True False False 159m
service-catalog-controller-manager 4.4.9 True False False 158m
storage 4.4.9 True False False 52m
[ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.5.0-0.nightly-2020-07-25-031342 True False 24m Cluster version is 4.5.0-0.nightly-2020-07-25-031342
[ramakasturinarra@dhcp35-60 ~]$ oc get co kube-scheduler
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
kube-scheduler 4.5.0-0.nightly-2020-07-25-031342 True False False 127m
[ramakasturinarra@dhcp35-60 ~]$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.5.0-0.nightly-2020-07-25-031342 True False False 3h41m
cloud-credential 4.5.0-0.nightly-2020-07-25-031342 True False False 3h58m
cluster-autoscaler 4.5.0-0.nightly-2020-07-25-031342 True False False 3h48m
config-operator 4.5.0-0.nightly-2020-07-25-031342 True False False 64m
console 4.5.0-0.nightly-2020-07-25-031342 True False False 29m
csi-snapshot-controller 4.5.0-0.nightly-2020-07-25-031342 True False False 33m
dns 4.5.0-0.nightly-2020-07-25-031342 True False False 3h52m
etcd 4.5.0-0.nightly-2020-07-25-031342 True False False 130m
image-registry 4.5.0-0.nightly-2020-07-25-031342 True False False 34m
ingress 4.5.0-0.nightly-2020-07-25-031342 True False False 97m
insights 4.5.0-0.nightly-2020-07-25-031342 True False False 3h49m
kube-apiserver 4.5.0-0.nightly-2020-07-25-031342 True False False 3h52m
kube-controller-manager 4.5.0-0.nightly-2020-07-25-031342 True False False 128m
kube-scheduler 4.5.0-0.nightly-2020-07-25-031342 True False False 128m
kube-storage-version-migrator 4.5.0-0.nightly-2020-07-25-031342 True False False 33m
machine-api 4.5.0-0.nightly-2020-07-25-031342 True False False 3h53m
machine-approver 4.5.0-0.nightly-2020-07-25-031342 True False False 57m
machine-config 4.5.0-0.nightly-2020-07-25-031342 True False False 3h52m
marketplace 4.5.0-0.nightly-2020-07-25-031342 True False False 29m
monitoring 4.5.0-0.nightly-2020-07-25-031342 True False False 55m
network 4.5.0-0.nightly-2020-07-25-031342 True False False 3h53m
node-tuning 4.5.0-0.nightly-2020-07-25-031342 True False False 58m
openshift-apiserver 4.5.0-0.nightly-2020-07-25-031342 True False False 104m
openshift-controller-manager 4.5.0-0.nightly-2020-07-25-031342 True False False 55m
openshift-samples 4.5.0-0.nightly-2020-07-25-031342 True False False 57m
operator-lifecycle-manager 4.5.0-0.nightly-2020-07-25-031342 True False False 3h49m
operator-lifecycle-manager-catalog 4.5.0-0.nightly-2020-07-25-031342 True False False 3h49m
operator-lifecycle-manager-packageserver 4.5.0-0.nightly-2020-07-25-031342 True False False 35m
service-ca 4.5.0-0.nightly-2020-07-25-031342 True False False 3h53m
storage 4.5.0-0.nightly-2020-07-25-031342 True False False 58m
[ramakasturinarra@dhcp35-60 ~]$ oc logs -f openshift-kube-scheduler-ip-10-0-132-202.us-east-2.compute.internal -n openshift-kube-scheduler | grep "predicates"
I0727 09:19:18.315546 1 factory.go:291] Creating scheduler with fit predicates 'map[CheckNodeUnschedulable:{} CheckVolumeBinding:{} GeneralPredicates:{} MatchInterPodAffinity:{} MaxAzureDiskVolumeCount:{} MaxCSIVolumeCountPred:{} MaxEBSVolumeCount:{} MaxGCEPDVolumeCount:{} NoDiskConflict:{} NoVolumeZoneConflict:{} PodToleratesNodeTaints:{}]' and priority functions 'map[BalancedResourceAllocation:1 ImageLocalityPriority:1 InterPodAffinityPriority:1 LeastRequestedPriority:1 NodeAffinityPriority:1 NodePreferAvoidPodsPriority:1 SelectorSpreadPriority:1 TaintTolerationPriority:1]'
[ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.6.0-0.nightly-2020-07-25-091217 True False 7m17s Cluster version is 4.6.0-0.nightly-2020-07-25-091217
[ramakasturinarra@dhcp35-60 ~]$ oc get co kube-scheduler
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
kube-scheduler 4.6.0-0.nightly-2020-07-25-091217 True False False 3h46m
[ramakasturinarra@dhcp35-60 ~]$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.6.0-0.nightly-2020-07-25-091217 True False False 12m
cloud-credential 4.6.0-0.nightly-2020-07-25-091217 True False False 5h37m
cluster-autoscaler 4.6.0-0.nightly-2020-07-25-091217 True False False 5h27m
config-operator 4.6.0-0.nightly-2020-07-25-091217 True False False 164m
console 4.6.0-0.nightly-2020-07-25-091217 True False False 19m
csi-snapshot-controller 4.6.0-0.nightly-2020-07-25-091217 True False False 132m
dns 4.6.0-0.nightly-2020-07-25-091217 True False False 5h31m
etcd 4.6.0-0.nightly-2020-07-25-091217 True False False 3h50m
image-registry 4.6.0-0.nightly-2020-07-25-091217 True False False 16m
ingress 4.6.0-0.nightly-2020-07-25-091217 True False False 83m
insights 4.6.0-0.nightly-2020-07-25-091217 True False False 5h28m
kube-apiserver 4.6.0-0.nightly-2020-07-25-091217 True False False 5h31m
kube-controller-manager 4.6.0-0.nightly-2020-07-25-091217 True False False 3h47m
kube-scheduler 4.6.0-0.nightly-2020-07-25-091217 True False False 3h47m
kube-storage-version-migrator 4.6.0-0.nightly-2020-07-25-091217 True False False 16m
machine-api 4.6.0-0.nightly-2020-07-25-091217 True False False 5h32m
machine-approver 4.6.0-0.nightly-2020-07-25-091217 True False False 157m
machine-config 4.6.0-0.nightly-2020-07-25-091217 True False False 10m
marketplace 4.6.0-0.nightly-2020-07-25-091217 True False False 14m
monitoring 4.6.0-0.nightly-2020-07-25-091217 True False False 20m
network 4.6.0-0.nightly-2020-07-25-091217 True False False 5h33m
node-tuning 4.6.0-0.nightly-2020-07-25-091217 True False False 83m
openshift-apiserver 4.6.0-0.nightly-2020-07-25-091217 True False False 3h23m
openshift-controller-manager 4.6.0-0.nightly-2020-07-25-091217 True False False 154m
openshift-samples 4.6.0-0.nightly-2020-07-25-091217 True False False 83m
operator-lifecycle-manager 4.6.0-0.nightly-2020-07-25-091217 True False False 5h28m
operator-lifecycle-manager-catalog 4.6.0-0.nightly-2020-07-25-091217 True False False 5h28m
operator-lifecycle-manager-packageserver 4.6.0-0.nightly-2020-07-25-091217 True False False 14m
service-ca 4.6.0-0.nightly-2020-07-25-091217 True False False 5h32m
storage 4.6.0-0.nightly-2020-07-25-091217 True False False 83m
[ramakasturinarra@dhcp35-60 ~]$ oc logs -f openshift-kube-scheduler-ip-10-0-132-202.us-east-2.compute.internal -c kube-scheduler -n openshift-kube-scheduler | grep predicates
I0727 11:16:28.826835 1 factory.go:279] Creating scheduler with fit predicates 'map[CheckNodeUnschedulable:{} CheckVolumeBinding:{} GeneralPredicates:{} MatchInterPodAffinity:{} MaxAzureDiskVolumeCount:{} MaxCSIVolumeCountPred:{} MaxEBSVolumeCount:{} MaxGCEPDVolumeCount:{} NoDiskConflict:{} NoVolumeZoneConflict:{} PodToleratesNodeTaints:{}]' and priority functions 'map[BalancedResourceAllocation:1 ImageLocalityPriority:1 InterPodAffinityPriority:1 LeastRequestedPriority:1 NodeAffinityPriority:1 NodePreferAvoidPodsPriority:1 SelectorSpreadPriority:1 TaintTolerationPriority:1]'
Based on the above data moving the bug to verified state.
*** Bug 1879355 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |