Bug 2041985 - When upgrading from 4.9 to 4.10 kubescheduler in terminated state with error tes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)\nStaticPodsDegraded
Summary: When upgrading from 4.9 to 4.10 kubescheduler in terminated state with error ...
Keywords:
Status: CLOSED DUPLICATE of bug 2037665
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-scheduler
Version: 4.10
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: Jan Chaloupka
QA Contact: RamaKasturi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-18 16:32 UTC by RamaKasturi
Modified: 2022-01-26 12:23 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-01-26 12:21:28 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-kube-scheduler-operator pull 400 0 None Merged bug 2037665: Set upgradeable=false if using Policy field 2022-01-25 15:03:53 UTC
Github openshift cluster-kube-scheduler-operator pull 402 0 None Merged bug 2037665: Remove policy upgradeable when policy field cleared 2022-01-25 15:03:53 UTC
Github openshift cluster-kube-scheduler-operator pull 403 0 None Merged bug 2037665: Check policy name field independently of the configmap change 2022-01-25 15:03:53 UTC

Description RamaKasturi 2022-01-18 16:32:17 UTC
Description of problem:
When upgrading from 4.9 to 4.10 kube-scheduler is in terminated state with errors below tes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)\nStaticPodsDegraded

01-18 03:03:43.377        [36m  - lastTransitionTime: "2022-01-17T19:50:18Z"[0m
01-18 03:03:43.377        [36m    message: "StaticPodsDegraded: pod/openshift-kube-scheduler-ip-10-0-151-230.us-east-2.compute.internal[0m
01-18 03:03:43.377        [36m      container \"kube-scheduler\" is terminated: Error: tes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)\nStaticPodsDegraded:[0m
01-18 03:03:43.377        [36m      \      --log-file-max-size uint           Defines the maximum size a log file[0m
01-18 03:03:43.377        [36m      can grow to. Unit is megabytes. If the value is 0, the maximum file size is[0m
01-18 03:03:43.377        [36m      unlimited. (default 1800) (DEPRECATED: will be removed in a future release,[0m
01-18 03:03:43.377        [36m      see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)\nStaticPodsDegraded:[0m
01-18 03:03:43.377        [36m      \      --logtostderr                      log to standard error instead of files[0m
01-18 03:03:43.377        [36m      (default true) (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)\nStaticPodsDegraded:[0m
01-18 03:03:43.377        [36m      \      --one-output                       If true, only write logs to their[0m
01-18 03:03:43.377        [36m      native severity level (vs also writing to each lower severity level) (DEPRECATED:[0m
01-18 03:03:43.377        [36m      will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)\nStaticPodsDegraded:[0m
01-18 03:03:43.377        [36m      \      --skip-headers                     If true, avoid header prefixes in[0m
01-18 03:03:43.377        [36m      the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)\nStaticPodsDegraded:[0m
01-18 03:03:43.377        [36m      \      --skip-log-headers                 If true, avoid headers when opening[0m
01-18 03:03:43.377        [36m      log files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)\nStaticPodsDegraded:[0m
01-18 03:03:43.377        [36m      \      --stderrthreshold severity         logs at or above this threshold go[0m
01-18 03:03:43.377        [36m      to stderr (default 2) (DEPRECATED: will be removed in a future release, see[0m
01-18 03:03:43.377        [36m      https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)\nStaticPodsDegraded:[0m
01-18 03:03:43.377        [36m      \      --version version[=true]           Print version information and quit\nStaticPodsDegraded:[0m
01-18 03:03:43.377        [36m      \nStaticPodsDegraded: Error: unknown flag: --policy-configmap\nNodeControllerDegraded:[0m

Version-Release number of selected component (if applicable):
4.9.0-0.nightly-2022-01-17-165424 -> 4.10.0-0.nightly-2022-01-17-182202

How reproducible:
Always

Steps to Reproduce:
1. Install latest 4.9 nightly
2. Create policy.cfg with the contents below
create a policy.cfg file with the contents below
{
        "kind" : "Policy",
        "apiVersion" : "v1",
        "predicates" : [
                {"name" : "MaxGCEPDVolumeCount"},
                {"name" : "GeneralPredicates"}, 
                {"name" : "MaxAzureDiskVolumeCount"},
                {"name" : "MaxCSIVolumeCountPred"},
                {"name" : "CheckVolumeBinding"},
                {"name" : "MaxEBSVolumeCount"},
                {"name" : "MatchInterPodAffinity"},
                {"name" : "CheckNodeUnschedulable"},
                {"name" : "NoDiskConflict"},
                {"name" : "NoVolumeZoneConflict"},
                {"name" : "PodToleratesNodeTaints"}
                ],
        "priorities" : [
                {"name" : "LeastRequestedPriority", "weight" : 1},
                {"name" : "BalancedResourceAllocation", "weight" : 1},
                {"name" : "ServiceSpreadingPriority", "weight" : 1},
                {"name" : "NodePreferAvoidPodsPriority", "weight" : 1},
                {"name" : "NodeAffinityPriority", "weight" : 1},
                {"name" : "TaintTolerationPriority", "weight" : 1},
                {"name" : "ImageLocalityPriority", "weight" : 1},
                {"name" : "SelectorSpreadPriority", "weight" : 1},
                {"name" : "InterPodAffinityPriority", "weight" : 1},
                {"name" : "EqualPriority", "weight" : 1}
                ]
    }
3.oc create configmap -n openshift-config --from-file=policy.cfg scheduler-policy
4. oc patch Scheduler cluster --type='merge' -p '{"spec":{"policy":{"name":"scheduler-policy"}}}' --type=merge
5. Run the upgrade command to perform an upgrade from 4.9 to 4.10

Actual results:
kube-scheduler is in terminated state with errors as stated in the description

Expected results:
kube-scheduler should not be in terminated state and upgrade should work fine.

Additional info:
I have another upgrade bug (https://bugzilla.redhat.com/show_bug.cgi?id=2037665)with similar steps but errors seen here are different than what we have seen in the other bug.

Comment 2 Jan Chaloupka 2022-01-20 15:10:50 UTC

*** This bug has been marked as a duplicate of bug 2037665 ***

Comment 3 Jan Chaloupka 2022-01-20 15:12:03 UTC
Closing the as duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2037665 based on the following error message:
```
StaticPodsDegraded: Error: unknown flag: --policy-configmap
```

Comment 4 Roshni 2022-01-25 12:26:13 UTC
I am re-opening this bug because it is blocking QE's 4.10 upgrade tests on ci and we do not have a 4.10 bug to track this.

Comment 5 Roshni 2022-01-26 12:21:28 UTC

*** This bug has been marked as a duplicate of bug 2037665 ***

Comment 6 Roshni 2022-01-26 12:23:40 UTC
Closing again because QE's upgrade tests on ci works fine after disabling one of the tests which is related to https://bugzilla.redhat.com/show_bug.cgi?id=2037665


Note You need to log in before you can comment on or make changes to this bug.