1840616 – Upgrading a cluster from 4.3.18 to 4.4.4 did not works when using a custom policy for kube-scheduler

Bug 1840616 - Upgrading a cluster from 4.3.18 to 4.4.4 did not works when using a custom policy for kube-scheduler

Summary: Upgrading a cluster from 4.3.18 to 4.4.4 did not works when using a custom po...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	kube-scheduler
Sub Component:
Version:	4.4
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	4.6.0
Assignee:	Jan Chaloupka
QA Contact:	RamaKasturi
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1879355 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-05-27 10:34 UTC by Andreas Nowak
Modified:	2020-10-27 16:01 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-10-27 16:01:07 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift openshift-docs pull 23346	0	None	closed	bug 1840616: kube-scheduler custom policy: do not use GeneralPredicates alongside its representants	2021-01-15 11:09:44 UTC
Red Hat Product Errata	RHBA-2020:4196	0	None	None	None	2020-10-27 16:01:21 UTC

Comment 14 RamaKasturi 2020-06-24 06:05:51 UTC

@jan, one question if this a documentation related bug, why do not we change the component as well to the documentation ?

Comment 15 Jan Chaloupka 2020-07-09 10:37:19 UTC

Updated Docs PR: https://github.com/openshift/openshift-docs/pull/23346

> @jan, one question if this a documentation related bug, why do not we change the component as well to the documentation ?

Changing the component will make harder to find the issue again.

Comment 19 RamaKasturi 2020-07-27 12:31:56 UTC

Tried to verify the bug by upgrading from 4.3.18 -> 4.4.9 -> 4.5 nightly -> 4.6. Was able to successfully upgrade by having the mentioned settings in the policy.cfg file and did not hit any issue. No issues reported from kube-scheduler as well.

[ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.18    True        False         82m     Cluster version is 4.3.18


[ramakasturinarra@dhcp35-60 ~]$ oc logs -f openshift-kube-scheduler-ip-10-0-132-202.us-east-2.compute.internal -n openshift-kube-scheduler | grep predicates
I0727 07:13:27.918092       1 factory.go:382] Creating scheduler with fit predicates 'map[CheckNodeUnschedulable:{} CheckVolumeBinding:{} GeneralPredicates:{} MatchInterPodAffinity:{} MaxAzureDiskVolumeCount:{} MaxCSIVolumeCountPred:{} MaxEBSVolumeCount:{} MaxGCEPDVolumeCount:{} NoDiskConflict:{} NoVolumeZoneConflict:{} PodToleratesNodeTaints:{}]' and priority functions 'map[BalancedResourceAllocation:{} EqualPriority:{} ImageLocalityPriority:{} InterPodAffinityPriority:{} LeastRequestedPriority:{} NodeAffinityPriority:{} NodePreferAvoidPodsPriority:{} SelectorSpreadPriority:{} ServiceSpreadingPriority:{} TaintTolerationPriority:{}]'

[ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.9     True        False         17m     Cluster version is 4.4.9

[ramakasturinarra@dhcp35-60 ~]$ oc get co kube-scheduler
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
kube-scheduler   4.4.9     True        False         False      57m

[ramakasturinarra@dhcp35-60 ~]$ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.4.9     True        False         False      150m
cloud-credential                           4.4.9     True        False         False      167m
cluster-autoscaler                         4.4.9     True        False         False      157m
console                                    4.4.9     True        False         False      24m
csi-snapshot-controller                    4.4.9     True        False         False      25m
dns                                        4.4.9     True        False         False      162m
etcd                                       4.4.9     True        False         False      60m
image-registry                             4.4.9     True        False         False      27m
ingress                                    4.4.9     True        False         False      26m
insights                                   4.4.9     True        False         False      158m
kube-apiserver                             4.4.9     True        False         False      161m
kube-controller-manager                    4.4.9     True        False         False      57m
kube-scheduler                             4.4.9     True        False         False      57m
kube-storage-version-migrator              4.4.9     True        False         False      27m
machine-api                                4.4.9     True        False         False      162m
machine-config                             4.4.9     True        False         False      162m
marketplace                                4.4.9     True        False         False      28m
monitoring                                 4.4.9     True        False         False      58m
network                                    4.4.9     True        False         False      163m
node-tuning                                4.4.9     True        False         False      7m59s
openshift-apiserver                        4.4.9     True        False         False      34m
openshift-controller-manager               4.4.9     True        False         False      161m
openshift-samples                          4.4.9     True        False         False      51m
operator-lifecycle-manager                 4.4.9     True        False         False      158m
operator-lifecycle-manager-catalog         4.4.9     True        False         False      158m
operator-lifecycle-manager-packageserver   4.4.9     True        False         False      24m
service-ca                                 4.4.9     True        False         False      163m
service-catalog-apiserver                  4.4.9     True        False         False      159m
service-catalog-controller-manager         4.4.9     True        False         False      158m
storage                                    4.4.9     True        False         False      52m

[ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-07-25-031342   True        False         24m     Cluster version is 4.5.0-0.nightly-2020-07-25-031342


[ramakasturinarra@dhcp35-60 ~]$ oc get co kube-scheduler
NAME             VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
kube-scheduler   4.5.0-0.nightly-2020-07-25-031342   True        False         False      127m
[ramakasturinarra@dhcp35-60 ~]$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.5.0-0.nightly-2020-07-25-031342   True        False         False      3h41m
cloud-credential                           4.5.0-0.nightly-2020-07-25-031342   True        False         False      3h58m
cluster-autoscaler                         4.5.0-0.nightly-2020-07-25-031342   True        False         False      3h48m
config-operator                            4.5.0-0.nightly-2020-07-25-031342   True        False         False      64m
console                                    4.5.0-0.nightly-2020-07-25-031342   True        False         False      29m
csi-snapshot-controller                    4.5.0-0.nightly-2020-07-25-031342   True        False         False      33m
dns                                        4.5.0-0.nightly-2020-07-25-031342   True        False         False      3h52m
etcd                                       4.5.0-0.nightly-2020-07-25-031342   True        False         False      130m
image-registry                             4.5.0-0.nightly-2020-07-25-031342   True        False         False      34m
ingress                                    4.5.0-0.nightly-2020-07-25-031342   True        False         False      97m
insights                                   4.5.0-0.nightly-2020-07-25-031342   True        False         False      3h49m
kube-apiserver                             4.5.0-0.nightly-2020-07-25-031342   True        False         False      3h52m
kube-controller-manager                    4.5.0-0.nightly-2020-07-25-031342   True        False         False      128m
kube-scheduler                             4.5.0-0.nightly-2020-07-25-031342   True        False         False      128m
kube-storage-version-migrator              4.5.0-0.nightly-2020-07-25-031342   True        False         False      33m
machine-api                                4.5.0-0.nightly-2020-07-25-031342   True        False         False      3h53m
machine-approver                           4.5.0-0.nightly-2020-07-25-031342   True        False         False      57m
machine-config                             4.5.0-0.nightly-2020-07-25-031342   True        False         False      3h52m
marketplace                                4.5.0-0.nightly-2020-07-25-031342   True        False         False      29m
monitoring                                 4.5.0-0.nightly-2020-07-25-031342   True        False         False      55m
network                                    4.5.0-0.nightly-2020-07-25-031342   True        False         False      3h53m
node-tuning                                4.5.0-0.nightly-2020-07-25-031342   True        False         False      58m
openshift-apiserver                        4.5.0-0.nightly-2020-07-25-031342   True        False         False      104m
openshift-controller-manager               4.5.0-0.nightly-2020-07-25-031342   True        False         False      55m
openshift-samples                          4.5.0-0.nightly-2020-07-25-031342   True        False         False      57m
operator-lifecycle-manager                 4.5.0-0.nightly-2020-07-25-031342   True        False         False      3h49m
operator-lifecycle-manager-catalog         4.5.0-0.nightly-2020-07-25-031342   True        False         False      3h49m
operator-lifecycle-manager-packageserver   4.5.0-0.nightly-2020-07-25-031342   True        False         False      35m
service-ca                                 4.5.0-0.nightly-2020-07-25-031342   True        False         False      3h53m
storage                                    4.5.0-0.nightly-2020-07-25-031342   True        False         False      58m

[ramakasturinarra@dhcp35-60 ~]$ oc logs -f openshift-kube-scheduler-ip-10-0-132-202.us-east-2.compute.internal -n openshift-kube-scheduler | grep "predicates"
I0727 09:19:18.315546       1 factory.go:291] Creating scheduler with fit predicates 'map[CheckNodeUnschedulable:{} CheckVolumeBinding:{} GeneralPredicates:{} MatchInterPodAffinity:{} MaxAzureDiskVolumeCount:{} MaxCSIVolumeCountPred:{} MaxEBSVolumeCount:{} MaxGCEPDVolumeCount:{} NoDiskConflict:{} NoVolumeZoneConflict:{} PodToleratesNodeTaints:{}]' and priority functions 'map[BalancedResourceAllocation:1 ImageLocalityPriority:1 InterPodAffinityPriority:1 LeastRequestedPriority:1 NodeAffinityPriority:1 NodePreferAvoidPodsPriority:1 SelectorSpreadPriority:1 TaintTolerationPriority:1]'



[ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-07-25-091217   True        False         7m17s   Cluster version is 4.6.0-0.nightly-2020-07-25-091217

[ramakasturinarra@dhcp35-60 ~]$ oc get co kube-scheduler
NAME             VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
kube-scheduler   4.6.0-0.nightly-2020-07-25-091217   True        False         False      3h46m

[ramakasturinarra@dhcp35-60 ~]$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.6.0-0.nightly-2020-07-25-091217   True        False         False      12m
cloud-credential                           4.6.0-0.nightly-2020-07-25-091217   True        False         False      5h37m
cluster-autoscaler                         4.6.0-0.nightly-2020-07-25-091217   True        False         False      5h27m
config-operator                            4.6.0-0.nightly-2020-07-25-091217   True        False         False      164m
console                                    4.6.0-0.nightly-2020-07-25-091217   True        False         False      19m
csi-snapshot-controller                    4.6.0-0.nightly-2020-07-25-091217   True        False         False      132m
dns                                        4.6.0-0.nightly-2020-07-25-091217   True        False         False      5h31m
etcd                                       4.6.0-0.nightly-2020-07-25-091217   True        False         False      3h50m
image-registry                             4.6.0-0.nightly-2020-07-25-091217   True        False         False      16m
ingress                                    4.6.0-0.nightly-2020-07-25-091217   True        False         False      83m
insights                                   4.6.0-0.nightly-2020-07-25-091217   True        False         False      5h28m
kube-apiserver                             4.6.0-0.nightly-2020-07-25-091217   True        False         False      5h31m
kube-controller-manager                    4.6.0-0.nightly-2020-07-25-091217   True        False         False      3h47m
kube-scheduler                             4.6.0-0.nightly-2020-07-25-091217   True        False         False      3h47m
kube-storage-version-migrator              4.6.0-0.nightly-2020-07-25-091217   True        False         False      16m
machine-api                                4.6.0-0.nightly-2020-07-25-091217   True        False         False      5h32m
machine-approver                           4.6.0-0.nightly-2020-07-25-091217   True        False         False      157m
machine-config                             4.6.0-0.nightly-2020-07-25-091217   True        False         False      10m
marketplace                                4.6.0-0.nightly-2020-07-25-091217   True        False         False      14m
monitoring                                 4.6.0-0.nightly-2020-07-25-091217   True        False         False      20m
network                                    4.6.0-0.nightly-2020-07-25-091217   True        False         False      5h33m
node-tuning                                4.6.0-0.nightly-2020-07-25-091217   True        False         False      83m
openshift-apiserver                        4.6.0-0.nightly-2020-07-25-091217   True        False         False      3h23m
openshift-controller-manager               4.6.0-0.nightly-2020-07-25-091217   True        False         False      154m
openshift-samples                          4.6.0-0.nightly-2020-07-25-091217   True        False         False      83m
operator-lifecycle-manager                 4.6.0-0.nightly-2020-07-25-091217   True        False         False      5h28m
operator-lifecycle-manager-catalog         4.6.0-0.nightly-2020-07-25-091217   True        False         False      5h28m
operator-lifecycle-manager-packageserver   4.6.0-0.nightly-2020-07-25-091217   True        False         False      14m
service-ca                                 4.6.0-0.nightly-2020-07-25-091217   True        False         False      5h32m
storage                                    4.6.0-0.nightly-2020-07-25-091217   True        False         False      83m


[ramakasturinarra@dhcp35-60 ~]$ oc logs -f openshift-kube-scheduler-ip-10-0-132-202.us-east-2.compute.internal -c kube-scheduler -n openshift-kube-scheduler | grep predicates
I0727 11:16:28.826835       1 factory.go:279] Creating scheduler with fit predicates 'map[CheckNodeUnschedulable:{} CheckVolumeBinding:{} GeneralPredicates:{} MatchInterPodAffinity:{} MaxAzureDiskVolumeCount:{} MaxCSIVolumeCountPred:{} MaxEBSVolumeCount:{} MaxGCEPDVolumeCount:{} NoDiskConflict:{} NoVolumeZoneConflict:{} PodToleratesNodeTaints:{}]' and priority functions 'map[BalancedResourceAllocation:1 ImageLocalityPriority:1 InterPodAffinityPriority:1 LeastRequestedPriority:1 NodeAffinityPriority:1 NodePreferAvoidPodsPriority:1 SelectorSpreadPriority:1 TaintTolerationPriority:1]'


Based on the above data moving the bug to verified state.

Comment 20 Jan Chaloupka 2020-09-16 10:29:59 UTC

*** Bug 1879355 has been marked as a duplicate of this bug. ***

Comment 22 errata-xmlrpc 2020-10-27 16:01:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196

Note You need to log in before you can comment on or make changes to this bug.