Bug 2042049 - Seeing warning related to unrecognized feature gate in kubescheduler & KCM logs
Summary: Seeing warning related to unrecognized feature gate in kubescheduler & KCM logs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-scheduler
Version: 4.10
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.10.0
Assignee: Jan Chaloupka
QA Contact: RamaKasturi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-18 18:34 UTC by RamaKasturi
Modified: 2022-03-12 04:41 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-12 04:41:03 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift api pull 1100 0 None Merged bug 2042049: Delete feature gates removed from k8s.io/kubernetes/pkg/features/kube_features.go 2022-01-21 12:47:36 UTC
Github openshift cluster-kube-apiserver-operator pull 1298 0 None open bug 2042049: Delete feature gates removed from k8s.io/kubernetes/pkg/features/kube_features.go 2022-01-27 09:31:05 UTC
Github openshift cluster-kube-controller-manager-operator pull 592 0 None open bug 2042049: Delete feature gates removed from k8s.io/kubernetes/pkg/features/kube_features.go 2022-01-19 21:07:57 UTC
Github openshift cluster-kube-scheduler-operator pull 401 0 None open bug 2042049: Sync openshift/api 2022-01-19 20:48:30 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-12 04:41:20 UTC

Description RamaKasturi 2022-01-18 18:34:27 UTC
Description of problem:
I see warnings related to unrecognized feature gate in kubescheduler logs
[knarra@knarra cucushift]$ oc logs -f openshift-kube-scheduler-ip-10-0-134-32.us-east-2.compute.internal -n openshift-kube-scheduler
Using deprecated annotation `kubectl.kubernetes.io/default-logs-container` in pod/openshift-kube-scheduler-ip-10-0-134-32.us-east-2.compute.internal. Please use `kubectl.kubernetes.io/default-container` instead
W0118 16:18:04.481020       1 feature_gate.go:223] unrecognized feature gate: ServiceNodeExclusion
W0118 16:18:04.481090       1 feature_gate.go:223] unrecognized feature gate: LegacyNodeRoleBehavior
W0118 16:18:04.481093       1 feature_gate.go:223] unrecognized feature gate: NodeDisruptionExclusion
W0118 16:18:04.481098       1 feature_gate.go:223] unrecognized feature gate: SupportPodPidsLimit


Version-Release number of selected component (if applicable):
[knarra@knarra cucushift]$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-01-18-044014   True        False         127m    Cluster version is 4.10.0-0.nightly-2022-01-18-044014


How reproducible:
Always

Steps to Reproduce:
1. Install 4.10 cluster
2. Now run the command oc log -f <scheduler_pod> -n openshift-kube-scheduler
3.

Actual results:
Below warnings about unrecognized feature gates are seen in kubescheduler logs.
[knarra@knarra cucushift]$ oc logs -f openshift-kube-scheduler-ip-10-0-134-32.us-east-2.compute.internal -n openshift-kube-scheduler
Using deprecated annotation `kubectl.kubernetes.io/default-logs-container` in pod/openshift-kube-scheduler-ip-10-0-134-32.us-east-2.compute.internal. Please use `kubectl.kubernetes.io/default-container` instead
W0118 16:18:04.481020       1 feature_gate.go:223] unrecognized feature gate: ServiceNodeExclusion
W0118 16:18:04.481090       1 feature_gate.go:223] unrecognized feature gate: LegacyNodeRoleBehavior
W0118 16:18:04.481093       1 feature_gate.go:223] unrecognized feature gate: NodeDisruptionExclusion
W0118 16:18:04.481098       1 feature_gate.go:223] unrecognized feature gate: SupportPodPidsLimit


Expected results:
No warnings should be seen

Additional info:
Same warnins are seen in KCM logs as well
[knarra@knarra cucushift]$ oc logs -f kube-controller-manager-ip-10-0-134-32.us-east-2.compute.internal -n openshift-kube-controller-manager
+ timeout 3m /bin/bash -exuo pipefail -c 'while [ -n "$(ss -Htanop \( sport = 10257 \))" ]; do sleep 1; done'
++ ss -Htanop '(' sport = 10257 ')'
Copying system trust bundle
+ '[' -n '' ']'
+ '[' -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt ']'
+ echo 'Copying system trust bundle'
+ cp -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
+ '[' -f /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem ']'
+ exec hyperkube kube-controller-manager --openshift-config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml --kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --authentication-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --authorization-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt --requestheader-client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt -v=2 --tls-cert-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt --tls-private-key-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key --allocate-node-cidrs=false --cert-dir=/var/run/kubernetes --cloud-provider=aws --cluster-cidr=10.128.0.0/14 --cluster-name=knarra0118-wbsl9 --cluster-signing-cert-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.crt --cluster-signing-duration=720h --cluster-signing-key-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.key --configure-cloud-routes=false '--controllers=*' --controllers=-bootstrapsigner --controllers=-tokencleaner --controllers=-ttl --enable-dynamic-provisioning=true --feature-gates=APIPriorityAndFairness=true --feature-gates=CSIMigrationAWS=false --feature-gates=CSIMigrationAzureDisk=false --feature-gates=CSIMigrationAzureFile=false --feature-gates=CSIMigrationGCE=false --feature-gates=CSIMigrationOpenStack=false --feature-gates=CSIMigrationvSphere=false --feature-gates=DownwardAPIHugePages=true --feature-gates=LegacyNodeRoleBehavior=false --feature-gates=NodeDisruptionExclusion=true --feature-gates=PodSecurity=true --feature-gates=RotateKubeletServerCertificate=true --feature-gates=ServiceNodeExclusion=true --feature-gates=SupportPodPidsLimit=true --flex-volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --kube-api-burst=300 --kube-api-qps=150 --leader-elect-resource-lock=configmaps --leader-elect-retry-period=3s --leader-elect=true --pv-recycler-pod-template-filepath-hostpath=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml --pv-recycler-pod-template-filepath-nfs=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml --root-ca-file=/etc/kubernetes/static-pod-resources/configmaps/serviceaccount-ca/ca-bundle.crt --secure-port=10257 --service-account-private-key-file=/etc/kubernetes/static-pod-resources/secrets/service-account-private-key/service-account.key --service-cluster-ip-range=172.30.0.0/16 --use-service-account-credentials=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 --tls-min-version=VersionTLS12
W0118 16:21:18.036475       1 feature_gate.go:223] unrecognized feature gate: LegacyNodeRoleBehavior
W0118 16:21:18.036582       1 feature_gate.go:223] unrecognized feature gate: NodeDisruptionExclusion
W0118 16:21:18.036696       1 feature_gate.go:223] unrecognized feature gate: ServiceNodeExclusion
W0118 16:21:18.036735       1 feature_gate.go:223] unrecognized feature gate: SupportPodPidsLimit

Comment 1 Jan Chaloupka 2022-01-19 20:37:18 UTC
Based on https://github.com/kubernetes/kubernetes/pull/100776 LegacyNodeRoleBehavior, NodeDisruptionExclusion and ServiceNodeExclusion were GA-ed in 1.22.
Based on https://github.com/kubernetes/kubernetes/pull/104163 SupportPodPidsLimit was GA-ed in 1.20.

Comment 4 RamaKasturi 2022-01-27 09:49:42 UTC
Verified bug with the payload below and i see that there are no warnings present anymore in the logs.

kube-scheduler logs:
======================
[knarra@knarra ~]$ oc logs -f openshift-kube-scheduler-ip-10-0-130-58.ap-northeast-1.compute.internal -n openshift-kube-scheduler
Using deprecated annotation `kubectl.kubernetes.io/default-logs-container` in pod/openshift-kube-scheduler-ip-10-0-130-58.ap-northeast-1.compute.internal. Please use `kubectl.kubernetes.io/default-container` instead
I0127 06:54:39.651742       1 flags.go:64] FLAG: --add-dir-header="false"
I0127 06:54:39.651861       1 flags.go:64] FLAG: --address="127.0.0.1"
I0127 06:54:39.651868       1 flags.go:64] FLAG: --allow-metric-labels="[]"
I0127 06:54:39.651878       1 flags.go:64] FLAG: --alsologtostderr="false"
I0127 06:54:39.651883       1 flags.go:64] FLAG: --authentication-kubeconfig="/etc/kubernetes/static-pod-resources/configmaps/scheduler-kubeconfig/kubeconfig"
I0127 06:54:39.651891       1 flags.go:64] FLAG: --authentication-skip-lookup="false"
I0127 06:54:39.651905       1 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="10s"
I0127 06:54:39.651911       1 flags.go:64] FLAG: --authentication-tolerate-lookup-failure="true"
I0127 06:54:39.651915       1 flags.go:64] FLAG: --authorization-always-allow-paths="[/healthz,/readyz,/livez]"
I0127 06:54:39.651923       1 flags.go:64] FLAG: --authorization-kubeconfig="/etc/kubernetes/static-pod-resources/configmaps/scheduler-kubeconfig/kubeconfig"
I0127 06:54:39.651930       1 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
I0127 06:54:39.651934       1 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
I0127 06:54:39.651939       1 flags.go:64] FLAG: --bind-address="0.0.0.0"
I0127 06:54:39.651943       1 flags.go:64] FLAG: --cert-dir="/var/run/kubernetes"
I0127 06:54:39.651948       1 flags.go:64] FLAG: --client-ca-file=""
I0127 06:54:39.651951       1 flags.go:64] FLAG: --config="/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml"
I0127 06:54:39.651957       1 flags.go:64] FLAG: --contention-profiling="true"
I0127 06:54:39.651962       1 flags.go:64] FLAG: --disabled-metrics="[]"
I0127 06:54:39.651969       1 flags.go:64] FLAG: --experimental-logging-sanitization="false"
I0127 06:54:39.651973       1 flags.go:64] FLAG: --feature-gates="APIPriorityAndFairness=true,CSIMigrationAWS=false,CSIMigrationAzureDisk=false,CSIMigrationAzureFile=false,CSIMigrationGCE=false,CSIMigrationOpenStack=false,CSIMigrationvSphere=false,DownwardAPIHugePages=true,PodSecurity=true,RotateKubeletServerCertificate=true"
I0127 06:54:39.651999       1 flags.go:64] FLAG: --help="false"

kube-controller-manager logs:
====================================
[knarra@knarra ~]$ oc logs -f kube-controller-manager-ip-10-0-130-58.ap-northeast-1.compute.internal -n openshift-kube-controller-manager
+ timeout 3m /bin/bash -exuo pipefail -c 'while [ -n "$(ss -Htanop \( sport = 10257 \))" ]; do sleep 1; done'
++ ss -Htanop '(' sport = 10257 ')'
+ '[' -n '' ']'
+ '[' -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt ']'
+ echo 'Copying system trust bundle'
Copying system trust bundle
+ cp -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
+ '[' -f /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem ']'
+ exec hyperkube kube-controller-manager --openshift-config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml --kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --authentication-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --authorization-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt --requestheader-client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt -v=2 --tls-cert-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt --tls-private-key-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key --allocate-node-cidrs=false --cert-dir=/var/run/kubernetes --cloud-provider=aws --cluster-cidr=10.128.0.0/14 --cluster-name=knarra0127-phxxt --cluster-signing-cert-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.crt --cluster-signing-duration=720h --cluster-signing-key-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.key --configure-cloud-routes=false '--controllers=*' --controllers=-bootstrapsigner --controllers=-tokencleaner --controllers=-ttl --enable-dynamic-provisioning=true --feature-gates=APIPriorityAndFairness=true --feature-gates=CSIMigrationAWS=false --feature-gates=CSIMigrationAzureDisk=false --feature-gates=CSIMigrationAzureFile=false --feature-gates=CSIMigrationGCE=false --feature-gates=CSIMigrationOpenStack=false --feature-gates=CSIMigrationvSphere=false --feature-gates=DownwardAPIHugePages=true --feature-gates=PodSecurity=true --feature-gates=RotateKubeletServerCertificate=true --flex-volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --kube-api-burst=300 --kube-api-qps=150 --leader-elect-resource-lock=leases --leader-elect-retry-period=3s --leader-elect=true --pv-recycler-pod-template-filepath-hostpath=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml --pv-recycler-pod-template-filepath-nfs=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml --root-ca-file=/etc/kubernetes/static-pod-resources/configmaps/serviceaccount-ca/ca-bundle.crt --secure-port=10257 --service-account-private-key-file=/etc/kubernetes/static-pod-resources/secrets/service-account-private-key/service-account.key --service-cluster-ip-range=172.30.0.0/16 --use-service-account-credentials=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 --tls-min-version=VersionTLS12
I0127 06:58:21.022846       1 flags.go:64] FLAG: --add-dir-header="false"
I0127 06:58:21.022935       1 flags.go:64] FLAG: --address="127.0.0.1"
I0127 06:58:21.022942       1 flags.go:64] FLAG: --allocate-node-cidrs="false"
I0127 06:58:21.022948       1 flags.go:64] FLAG: --allow-metric-labels="[]"
I0127 06:58:21.022954       1 flags.go:64] FLAG: --allow-untagged-cloud="false"
I0127 06:58:21.022958       1 flags.go:64] FLAG: --alsologtostderr="false"
I0127 06:58:21.022963       1 flags.go:64] FLAG: --attach-detach-reconcile-sync-period="1m0s"
I0127 06:58:21.022969       1 flags.go:64] FLAG: --authentication-kubeconfig="/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig"
I0127 06:58:21.022984       1 flags.go:64] FLAG: --authentication-skip-lookup="false"
I0127 06:58:21.022988       1 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="10s"

kube-api-server logs:
=========================
checked the openshift-kube-api-server logs and i see that warnings are still present. so moving bug back to assigned state.

I0127 06:56:55.803934      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true]}
I0127 06:56:55.804011      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true RotateKubeletServerCertificate:true]}
W0127 06:56:55.804059      16 feature_gate.go:223] unrecognized feature gate: SupportPodPidsLimit
I0127 06:56:55.804065      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true RotateKubeletServerCertificate:true]}
W0127 06:56:55.804105      16 feature_gate.go:223] unrecognized feature gate: NodeDisruptionExclusion
I0127 06:56:55.804111      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true RotateKubeletServerCertificate:true]}
W0127 06:56:55.804151      16 feature_gate.go:223] unrecognized feature gate: ServiceNodeExclusion
I0127 06:56:55.804156      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true RotateKubeletServerCertificate:true]}
I0127 06:56:55.804196      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true DownwardAPIHugePages:true RotateKubeletServerCertificate:true]}
I0127 06:56:55.804235      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
W0127 06:56:55.804299      16 feature_gate.go:223] unrecognized feature gate: LegacyNodeRoleBehavior
I0127 06:56:55.804304      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0127 06:56:55.804343      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0127 06:56:55.804393      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false CSIMigrationOpenStack:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0127 06:56:55.804434      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false CSIMigrationGCE:false CSIMigrationOpenStack:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}

Based on the above moving bug to assigned state.

Comment 9 RamaKasturi 2022-01-31 09:01:14 UTC
Verified with the build below and i do not see any warnings related to feature gate present in ks, kcm & kas logs. 

kube-scheduler logs:
=========================
[knarra@knarra cucushift]$ oc logs -f openshift-kube-scheduler-ip-10-0-192-9.us-east-2.compute.internal -n openshift-kube-scheduler
I0131 07:28:49.685986       1 flags.go:64] FLAG: --add-dir-header="false"
I0131 07:28:49.686132       1 flags.go:64] FLAG: --address="127.0.0.1"
I0131 07:28:49.686138       1 flags.go:64] FLAG: --allow-metric-labels="[]"
I0131 07:28:49.686152       1 flags.go:64] FLAG: --alsologtostderr="false"
I0131 07:28:49.686155       1 flags.go:64] FLAG: --authentication-kubeconfig="/etc/kubernetes/static-pod-resources/configmaps/scheduler-kubeconfig/kubeconfig"
I0131 07:28:49.686161       1 flags.go:64] FLAG: --authentication-skip-lookup="false"
I0131 07:28:49.686165       1 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="10s"
I0131 07:28:49.686169       1 flags.go:64] FLAG: --authentication-tolerate-lookup-failure="true"
I0131 07:28:49.686172       1 flags.go:64] FLAG: --authorization-always-allow-paths="[/healthz,/readyz,/livez]"
I0131 07:28:49.686179       1 flags.go:64] FLAG: --authorization-kubeconfig="/etc/kubernetes/static-pod-resources/configmaps/scheduler-kubeconfig/kubeconfig"
I0131 07:28:49.686183       1 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
I0131 07:28:49.686186       1 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
I0131 07:28:49.686189       1 flags.go:64] FLAG: --bind-address="0.0.0.0"
I0131 07:28:49.686192       1 flags.go:64] FLAG: --cert-dir="/var/run/kubernetes"
I0131 07:28:49.686195       1 flags.go:64] FLAG: --client-ca-file=""
I0131 07:28:49.686198       1 flags.go:64] FLAG: --config="/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml"
I0131 07:28:49.686202       1 flags.go:64] FLAG: --contention-profiling="true"
I0131 07:28:49.686205       1 flags.go:64] FLAG: --disabled-metrics="[]"
I0131 07:28:49.686209       1 flags.go:64] FLAG: --experimental-logging-sanitization="false"
I0131 07:28:49.686212       1 flags.go:64] FLAG: --feature-gates="APIPriorityAndFairness=true,CSIMigrationAWS=false,CSIMigrationAzureDisk=false,CSIMigrationAzureFile=false,CSIMigrationGCE=false,CSIMigrationOpenStack=false,CSIMigrationvSphere=false,DownwardAPIHugePages=true,PodSecurity=true,RotateKubeletServerCertificate=true"
I0131 07:28:49.686235       1 flags.go:64] FLAG: --help="false"
I0131 07:28:49.686238       1 flags.go:64] FLAG: --http2-max-streams-per-connection="0"
I0131 07:28:49.686245       1 flags.go:64] FLAG: --kube-api-burst="100"
I0131 07:28:49.686250       1 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I0131 07:28:49.686254       1 flags.go:64] FLAG: --kube-api-qps="50"
I0131 07:28:49.686259       1 flags.go:64] FLAG: --kubeconfig=""
I0131 07:28:49.686262       1 flags.go:64] FLAG: --leader-elect="true"
I0131 07:28:49.686265       1 flags.go:64] FLAG: --leader-elect-lease-duration="15s"
I0131 07:28:49.686269       1 flags.go:64] FLAG: --leader-elect-renew-deadline="10s"
I0131 07:28:49.686272       1 flags.go:64] FLAG: --leader-elect-resource-lock="leases"
I0131 07:28:49.686276       1 flags.go:64] FLAG: --leader-elect-resource-name="kube-scheduler"
I0131 07:28:49.686279       1 flags.go:64] FLAG: --leader-elect-resource-namespace="kube-system"
I0131 07:28:49.686282       1 flags.go:64] FLAG: --leader-elect-retry-period="2s"
I0131 07:28:49.686286       1 flags.go:64] FLAG: --lock-object-name="kube-scheduler"
I0131 07:28:49.686290       1 flags.go:64] FLAG: --lock-object-namespace="kube-system"
I0131 07:28:49.686293       1 flags.go:64] FLAG: --log-backtrace-at=":0"
I0131 07:28:49.686296       1 flags.go:64] FLAG: --log-dir=""
I0131 07:28:49.686300       1 flags.go:64] FLAG: --log-file=""
I0131 07:28:49.686303       1 flags.go:64] FLAG: --log-file-max-size="1800"
I0131 07:28:49.686307       1 flags.go:64] FLAG: --log-flush-frequency="5s"
I0131 07:28:49.686310       1 flags.go:64] FLAG: --log-json-info-buffer-size="0"
I0131 07:28:49.686324       1 flags.go:64] FLAG: --log-json-split-stream="false"
I0131 07:28:49.686327       1 flags.go:64] FLAG: --logging-format="text"
I0131 07:28:49.686331       1 flags.go:64] FLAG: --logtostderr="true"
I0131 07:28:49.686334       1 flags.go:64] FLAG: --master=""
I0131 07:28:49.686337       1 flags.go:64] FLAG: --one-output="false"
I0131 07:28:49.686342       1 flags.go:64] FLAG: --permit-address-sharing="false"
I0131 07:28:49.686345       1 flags.go:64] FLAG: --permit-port-sharing="false"
I0131 07:28:49.686348       1 flags.go:64] FLAG: --port="0"
I0131 07:28:49.686351       1 flags.go:64] FLAG: --profiling="true"
I0131 07:28:49.686354       1 flags.go:64] FLAG: --requestheader-allowed-names="[]"
I0131 07:28:49.686358       1 flags.go:64] FLAG: --requestheader-client-ca-file=""
I0131 07:28:49.686362       1 flags.go:64] FLAG: --requestheader-extra-headers-prefix="[x-remote-extra-]"
I0131 07:28:49.686368       1 flags.go:64] FLAG: --requestheader-group-headers="[x-remote-group]"
I0131 07:28:49.686373       1 flags.go:64] FLAG: --requestheader-username-headers="[x-remote-user]"
I0131 07:28:49.686378       1 flags.go:64] FLAG: --secure-port="10259"
I0131 07:28:49.686381       1 flags.go:64] FLAG: --show-hidden-metrics-for-version=""
I0131 07:28:49.686384       1 flags.go:64] FLAG: --skip-headers="false"
I0131 07:28:49.686387       1 flags.go:64] FLAG: --skip-log-headers="false"
I0131 07:28:49.686390       1 flags.go:64] FLAG: --stderrthreshold="2"
I0131 07:28:49.686394       1 flags.go:64] FLAG: --tls-cert-file="/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt"
I0131 07:28:49.686398       1 flags.go:64] FLAG: --tls-cipher-suites="[TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256]"
I0131 07:28:49.686412       1 flags.go:64] FLAG: --tls-min-version="VersionTLS12"
I0131 07:28:49.686417       1 flags.go:64] FLAG: --tls-private-key-file="/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key"
I0131 07:28:49.686421       1 flags.go:64] FLAG: --tls-sni-cert-key="[]"
I0131 07:28:49.686426       1 flags.go:64] FLAG: --unsupported-kube-api-over-localhost="false"
I0131 07:28:49.686429       1 flags.go:64] FLAG: --v="2"
I0131 07:28:49.686435       1 flags.go:64] FLAG: --version="false"
I0131 07:28:49.686440       1 flags.go:64] FLAG: --vmodule=""
I0131 07:28:49.686445       1 flags.go:64] FLAG: --write-config-to=""
I0131 07:28:49.687568       1 dynamic_serving_content.go:112] "Loaded a new cert/key pair" name="serving-cert::/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt::/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key"
I0131 07:28:49.967606       1 requestheader_controller.go:244] Loaded a new request header values for RequestHeaderAuthRequestController

[knarra@knarra cucushift]$ oc logs openshift-kube-scheduler-ip-10-0-192-9.us-east-2.compute.internal -n openshift-kube-scheduler | grep "unrecognized feature gate"
[knarra@knarra cucushift]$ 

[knarra@knarra cucushift]$ oc logs openshift-kube-scheduler-ip-10-0-192-9.us-east-2.compute.internal -n openshift-kube-scheduler | grep "unrecognized"
[knarra@knarra cucushift]$ 



kube-controller-manager logs:
================================
[knarra@knarra cucushift]$ oc logs -f kube-controller-manager-ip-10-0-192-9.us-east-2.compute.internal -n openshift-kube-controller-manager
+ timeout 3m /bin/bash -exuo pipefail -c 'while [ -n "$(ss -Htanop \( sport = 10257 \))" ]; do sleep 1; done'
++ ss -Htanop '(' sport = 10257 ')'
+ '[' -n '' ']'
+ '[' -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt ']'
Copying system trust bundle
+ echo 'Copying system trust bundle'
+ cp -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
+ '[' -f /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem ']'
+ exec hyperkube kube-controller-manager --openshift-config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml --kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --authentication-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --authorization-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig --client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt --requestheader-client-ca-file=/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt -v=2 --tls-cert-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt --tls-private-key-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key --allocate-node-cidrs=false --cert-dir=/var/run/kubernetes --cloud-provider=aws --cluster-cidr=10.128.0.0/14 --cluster-name=knarra0131-zbwqr --cluster-signing-cert-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.crt --cluster-signing-duration=720h --cluster-signing-key-file=/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.key --configure-cloud-routes=false '--controllers=*' --controllers=-bootstrapsigner --controllers=-tokencleaner --controllers=-ttl --enable-dynamic-provisioning=true --feature-gates=APIPriorityAndFairness=true --feature-gates=CSIMigrationAWS=false --feature-gates=CSIMigrationAzureDisk=false --feature-gates=CSIMigrationAzureFile=false --feature-gates=CSIMigrationGCE=false --feature-gates=CSIMigrationOpenStack=false --feature-gates=CSIMigrationvSphere=false --feature-gates=DownwardAPIHugePages=true --feature-gates=PodSecurity=true --feature-gates=RotateKubeletServerCertificate=true --flex-volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --kube-api-burst=300 --kube-api-qps=150 --leader-elect-resource-lock=leases --leader-elect-retry-period=3s --leader-elect=true --pv-recycler-pod-template-filepath-hostpath=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml --pv-recycler-pod-template-filepath-nfs=/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml --root-ca-file=/etc/kubernetes/static-pod-resources/configmaps/serviceaccount-ca/ca-bundle.crt --secure-port=10257 --service-account-private-key-file=/etc/kubernetes/static-pod-resources/secrets/service-account-private-key/service-account.key --service-cluster-ip-range=172.30.0.0/16 --use-service-account-credentials=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 --tls-min-version=VersionTLS12
I0131 07:32:09.399916       1 flags.go:64] FLAG: --add-dir-header="false"
I0131 07:32:09.400009       1 flags.go:64] FLAG: --address="127.0.0.1"
I0131 07:32:09.400016       1 flags.go:64] FLAG: --allocate-node-cidrs="false"
I0131 07:32:09.400020       1 flags.go:64] FLAG: --allow-metric-labels="[]"
I0131 07:32:09.400025       1 flags.go:64] FLAG: --allow-untagged-cloud="false"
I0131 07:32:09.400028       1 flags.go:64] FLAG: --alsologtostderr="false"
I0131 07:32:09.400031       1 flags.go:64] FLAG: --attach-detach-reconcile-sync-period="1m0s"
I0131 07:32:09.400040       1 flags.go:64] FLAG: --authentication-kubeconfig="/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig"
I0131 07:32:09.400049       1 flags.go:64] FLAG: --authentication-skip-lookup="false"
I0131 07:32:09.400052       1 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="10s"
I0131 07:32:09.400055       1 flags.go:64] FLAG: --authentication-tolerate-lookup-failure="false"
I0131 07:32:09.400058       1 flags.go:64] FLAG: --authorization-always-allow-paths="[/healthz,/readyz,/livez]"
I0131 07:32:09.400067       1 flags.go:64] FLAG: --authorization-kubeconfig="/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig"
I0131 07:32:09.400072       1 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
I0131 07:32:09.400076       1 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
I0131 07:32:09.400078       1 flags.go:64] FLAG: --bind-address="0.0.0.0"
I0131 07:32:09.400081       1 flags.go:64] FLAG: --cert-dir="/var/run/kubernetes"
I0131 07:32:09.400085       1 flags.go:64] FLAG: --cidr-allocator-type="RangeAllocator"
I0131 07:32:09.400088       1 flags.go:64] FLAG: --client-ca-file="/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt"
I0131 07:32:09.400092       1 flags.go:64] FLAG: --cloud-config=""
I0131 07:32:09.400095       1 flags.go:64] FLAG: --cloud-provider="aws"
I0131 07:32:09.400098       1 flags.go:64] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
I0131 07:32:09.400105       1 flags.go:64] FLAG: --cluster-cidr="10.128.0.0/14"
I0131 07:32:09.400108       1 flags.go:64] FLAG: --cluster-name="knarra0131-zbwqr"
I0131 07:32:09.400112       1 flags.go:64] FLAG: --cluster-signing-cert-file="/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.crt"
I0131 07:32:09.400117       1 flags.go:64] FLAG: --cluster-signing-duration="720h0m0s"
I0131 07:32:09.400121       1 flags.go:64] FLAG: --cluster-signing-key-file="/etc/kubernetes/static-pod-certs/secrets/csr-signer/tls.key"
I0131 07:32:09.400125       1 flags.go:64] FLAG: --cluster-signing-kube-apiserver-client-cert-file=""
I0131 07:32:09.400128       1 flags.go:64] FLAG: --cluster-signing-kube-apiserver-client-key-file=""
I0131 07:32:09.400131       1 flags.go:64] FLAG: --cluster-signing-kubelet-client-cert-file=""
I0131 07:32:09.400134       1 flags.go:64] FLAG: --cluster-signing-kubelet-client-key-file=""
I0131 07:32:09.400137       1 flags.go:64] FLAG: --cluster-signing-kubelet-serving-cert-file=""
I0131 07:32:09.400140       1 flags.go:64] FLAG: --cluster-signing-kubelet-serving-key-file=""
I0131 07:32:09.400143       1 flags.go:64] FLAG: --cluster-signing-legacy-unknown-cert-file=""
I0131 07:32:09.400146       1 flags.go:64] FLAG: --cluster-signing-legacy-unknown-key-file=""
I0131 07:32:09.400149       1 flags.go:64] FLAG: --concurrent-deployment-syncs="5"
I0131 07:32:09.400165       1 flags.go:64] FLAG: --concurrent-endpoint-syncs="5"
I0131 07:32:09.400172       1 flags.go:64] FLAG: --concurrent-ephemeralvolume-syncs="5"
I0131 07:32:09.400175       1 flags.go:64] FLAG: --concurrent-gc-syncs="20"
I0131 07:32:09.400178       1 flags.go:64] FLAG: --concurrent-namespace-syncs="10"
I0131 07:32:09.400181       1 flags.go:64] FLAG: --concurrent-rc-syncs="5"
I0131 07:32:09.400185       1 flags.go:64] FLAG: --concurrent-replicaset-syncs="5"
I0131 07:32:09.400188       1 flags.go:64] FLAG: --concurrent-resource-quota-syncs="5"
I0131 07:32:09.400191       1 flags.go:64] FLAG: --concurrent-service-endpoint-syncs="5"
I0131 07:32:09.400194       1 flags.go:64] FLAG: --concurrent-service-syncs="1"
I0131 07:32:09.400197       1 flags.go:64] FLAG: --concurrent-serviceaccount-token-syncs="5"
I0131 07:32:09.400200       1 flags.go:64] FLAG: --concurrent-statefulset-syncs="5"
I0131 07:32:09.400203       1 flags.go:64] FLAG: --concurrent-ttl-after-finished-syncs="5"
I0131 07:32:09.400206       1 flags.go:64] FLAG: --configure-cloud-routes="false"
I0131 07:32:09.400209       1 flags.go:64] FLAG: --contention-profiling="false"
I0131 07:32:09.400213       1 flags.go:64] FLAG: --controller-start-interval="0s"
I0131 07:32:09.400216       1 flags.go:64] FLAG: --controllers="[*,-bootstrapsigner,-tokencleaner,-ttl]"
I0131 07:32:09.400221       1 flags.go:64] FLAG: --deleting-pods-burst="0"
I0131 07:32:09.400224       1 flags.go:64] FLAG: --deleting-pods-qps="0.1"
I0131 07:32:09.400231       1 flags.go:64] FLAG: --deployment-controller-sync-period="30s"
I0131 07:32:09.400235       1 flags.go:64] FLAG: --disable-attach-detach-reconcile-sync="false"
I0131 07:32:09.400238       1 flags.go:64] FLAG: --disabled-metrics="[]"
I0131 07:32:09.400244       1 flags.go:64] FLAG: --enable-dynamic-provisioning="true"
I0131 07:32:09.400247       1 flags.go:64] FLAG: --enable-garbage-collector="true"
I0131 07:32:09.400251       1 flags.go:64] FLAG: --enable-hostpath-provisioner="false"
I0131 07:32:09.400254       1 flags.go:64] FLAG: --enable-leader-migration="false"
I0131 07:32:09.400257       1 flags.go:64] FLAG: --enable-taint-manager="true"
I0131 07:32:09.400260       1 flags.go:64] FLAG: --endpoint-updates-batch-period="0s"
I0131 07:32:09.400264       1 flags.go:64] FLAG: --endpointslice-updates-batch-period="0s"
I0131 07:32:09.400267       1 flags.go:64] FLAG: --experimental-cluster-signing-duration="720h0m0s"
I0131 07:32:09.400270       1 flags.go:64] FLAG: --experimental-logging-sanitization="false"
I0131 07:32:09.400273       1 flags.go:64] FLAG: --external-cloud-volume-plugin=""
I0131 07:32:09.400276       1 flags.go:64] FLAG: --feature-gates="APIPriorityAndFairness=true,CSIMigrationAWS=false,CSIMigrationAzureDisk=false,CSIMigrationAzureFile=false,CSIMigrationGCE=false,CSIMigrationOpenStack=false,CSIMigrationvSphere=false,DownwardAPIHugePages=true,PodSecurity=true,RotateKubeletServerCertificate=true"
I0131 07:32:09.400298       1 flags.go:64] FLAG: --flex-volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
I0131 07:32:09.400304       1 flags.go:64] FLAG: --help="false"
I0131 07:32:09.400307       1 flags.go:64] FLAG: --horizontal-pod-autoscaler-cpu-initialization-period="5m0s"
I0131 07:32:09.400310       1 flags.go:64] FLAG: --horizontal-pod-autoscaler-downscale-delay="5m0s"
I0131 07:32:09.400314       1 flags.go:64] FLAG: --horizontal-pod-autoscaler-downscale-stabilization="5m0s"
I0131 07:32:09.400317       1 flags.go:64] FLAG: --horizontal-pod-autoscaler-initial-readiness-delay="30s"
I0131 07:32:09.400320       1 flags.go:64] FLAG: --horizontal-pod-autoscaler-sync-period="15s"
I0131 07:32:09.400323       1 flags.go:64] FLAG: --horizontal-pod-autoscaler-tolerance="0.1"
I0131 07:32:09.400328       1 flags.go:64] FLAG: --horizontal-pod-autoscaler-upscale-delay="3m0s"
I0131 07:32:09.400331       1 flags.go:64] FLAG: --http2-max-streams-per-connection="0"
I0131 07:32:09.400336       1 flags.go:64] FLAG: --insecure-experimental-approve-all-kubelet-csrs-for-group=""
I0131 07:32:09.400339       1 flags.go:64] FLAG: --kube-api-burst="300"
I0131 07:32:09.400342       1 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I0131 07:32:09.400347       1 flags.go:64] FLAG: --kube-api-qps="150"
I0131 07:32:09.400351       1 flags.go:64] FLAG: --kubeconfig="/etc/kubernetes/static-pod-resources/configmaps/controller-manager-kubeconfig/kubeconfig"
I0131 07:32:09.400356       1 flags.go:64] FLAG: --large-cluster-size-threshold="50"
I0131 07:32:09.400360       1 flags.go:64] FLAG: --leader-elect="true"
I0131 07:32:09.400363       1 flags.go:64] FLAG: --leader-elect-lease-duration="15s"
I0131 07:32:09.400367       1 flags.go:64] FLAG: --leader-elect-renew-deadline="10s"
I0131 07:32:09.400371       1 flags.go:64] FLAG: --leader-elect-resource-lock="leases"
I0131 07:32:09.400374       1 flags.go:64] FLAG: --leader-elect-resource-name="kube-controller-manager"
I0131 07:32:09.400378       1 flags.go:64] FLAG: --leader-elect-resource-namespace="kube-system"
I0131 07:32:09.400382       1 flags.go:64] FLAG: --leader-elect-retry-period="3s"
I0131 07:32:09.400386       1 flags.go:64] FLAG: --leader-migration-config=""
I0131 07:32:09.400389       1 flags.go:64] FLAG: --log-backtrace-at=":0"
I0131 07:32:09.400394       1 flags.go:64] FLAG: --log-dir=""
I0131 07:32:09.400398       1 flags.go:64] FLAG: --log-file=""
I0131 07:32:09.400401       1 flags.go:64] FLAG: --log-file-max-size="1800"
I0131 07:32:09.400405       1 flags.go:64] FLAG: --log-flush-frequency="5s"
I0131 07:32:09.400409       1 flags.go:64] FLAG: --log-json-info-buffer-size="0"
I0131 07:32:09.400416       1 flags.go:64] FLAG: --log-json-split-stream="false"
I0131 07:32:09.400420       1 flags.go:64] FLAG: --logging-format="text"
I0131 07:32:09.400423       1 flags.go:64] FLAG: --logtostderr="true"
I0131 07:32:09.400427       1 flags.go:64] FLAG: --master=""
I0131 07:32:09.400430       1 flags.go:64] FLAG: --max-endpoints-per-slice="100"
I0131 07:32:09.400433       1 flags.go:64] FLAG: --min-resync-period="12h0m0s"
I0131 07:32:09.400436       1 flags.go:64] FLAG: --mirroring-concurrent-service-endpoint-syncs="5"
I0131 07:32:09.400439       1 flags.go:64] FLAG: --mirroring-endpointslice-updates-batch-period="0s"
I0131 07:32:09.400442       1 flags.go:64] FLAG: --mirroring-max-endpoints-per-subset="1000"
I0131 07:32:09.400447       1 flags.go:64] FLAG: --namespace-sync-period="5m0s"
I0131 07:32:09.400451       1 flags.go:64] FLAG: --node-cidr-mask-size="0"
I0131 07:32:09.400454       1 flags.go:64] FLAG: --node-cidr-mask-size-ipv4="0"
I0131 07:32:09.400457       1 flags.go:64] FLAG: --node-cidr-mask-size-ipv6="0"
I0131 07:32:09.400460       1 flags.go:64] FLAG: --node-eviction-rate="0.1"
I0131 07:32:09.400463       1 flags.go:64] FLAG: --node-monitor-grace-period="40s"
I0131 07:32:09.400467       1 flags.go:64] FLAG: --node-monitor-period="5s"
I0131 07:32:09.400470       1 flags.go:64] FLAG: --node-startup-grace-period="1m0s"
I0131 07:32:09.400473       1 flags.go:64] FLAG: --node-sync-period="0s"
I0131 07:32:09.400476       1 flags.go:64] FLAG: --one-output="false"
I0131 07:32:09.400479       1 flags.go:64] FLAG: --openshift-config="/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml"
I0131 07:32:09.400484       1 flags.go:64] FLAG: --permit-address-sharing="false"
I0131 07:32:09.400487       1 flags.go:64] FLAG: --permit-port-sharing="false"
I0131 07:32:09.400490       1 flags.go:64] FLAG: --pod-eviction-timeout="5m0s"
I0131 07:32:09.400521       1 flags.go:64] FLAG: --port="0"
I0131 07:32:09.400528       1 flags.go:64] FLAG: --profiling="true"
I0131 07:32:09.400531       1 flags.go:64] FLAG: --pv-recycler-increment-timeout-nfs="30"
I0131 07:32:09.400534       1 flags.go:64] FLAG: --pv-recycler-minimum-timeout-hostpath="60"
I0131 07:32:09.400537       1 flags.go:64] FLAG: --pv-recycler-minimum-timeout-nfs="300"
I0131 07:32:09.400554       1 flags.go:64] FLAG: --pv-recycler-pod-template-filepath-hostpath="/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml"
I0131 07:32:09.400559       1 flags.go:64] FLAG: --pv-recycler-pod-template-filepath-nfs="/etc/kubernetes/static-pod-resources/configmaps/recycler-config/recycler-pod.yaml"
I0131 07:32:09.400573       1 flags.go:64] FLAG: --pv-recycler-timeout-increment-hostpath="30"
I0131 07:32:09.400580       1 flags.go:64] FLAG: --pvclaimbinder-sync-period="15s"
I0131 07:32:09.400583       1 flags.go:64] FLAG: --register-retry-count="10"
I0131 07:32:09.400587       1 flags.go:64] FLAG: --requestheader-allowed-names="[]"
I0131 07:32:09.400591       1 flags.go:64] FLAG: --requestheader-client-ca-file="/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"
I0131 07:32:09.400595       1 flags.go:64] FLAG: --requestheader-extra-headers-prefix="[x-remote-extra-]"
I0131 07:32:09.400601       1 flags.go:64] FLAG: --requestheader-group-headers="[x-remote-group]"
I0131 07:32:09.400606       1 flags.go:64] FLAG: --requestheader-username-headers="[x-remote-user]"
I0131 07:32:09.400613       1 flags.go:64] FLAG: --resource-quota-sync-period="5m0s"
I0131 07:32:09.400616       1 flags.go:64] FLAG: --root-ca-file="/etc/kubernetes/static-pod-resources/configmaps/serviceaccount-ca/ca-bundle.crt"
I0131 07:32:09.400621       1 flags.go:64] FLAG: --route-reconciliation-period="10s"
I0131 07:32:09.400624       1 flags.go:64] FLAG: --secondary-node-eviction-rate="0.01"
I0131 07:32:09.400629       1 flags.go:64] FLAG: --secure-port="10257"
I0131 07:32:09.400632       1 flags.go:64] FLAG: --service-account-private-key-file="/etc/kubernetes/static-pod-resources/secrets/service-account-private-key/service-account.key"
I0131 07:32:09.400637       1 flags.go:64] FLAG: --service-cluster-ip-range="172.30.0.0/16"
I0131 07:32:09.400640       1 flags.go:64] FLAG: --show-hidden-metrics-for-version=""
I0131 07:32:09.400643       1 flags.go:64] FLAG: --skip-headers="false"
I0131 07:32:09.400647       1 flags.go:64] FLAG: --skip-log-headers="false"
I0131 07:32:09.400652       1 flags.go:64] FLAG: --stderrthreshold="2"
I0131 07:32:09.400655       1 flags.go:64] FLAG: --terminated-pod-gc-threshold="12500"
I0131 07:32:09.400658       1 flags.go:64] FLAG: --tls-cert-file="/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt"
I0131 07:32:09.400663       1 flags.go:64] FLAG: --tls-cipher-suites="[TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256]"
I0131 07:32:09.400678       1 flags.go:64] FLAG: --tls-min-version="VersionTLS12"
I0131 07:32:09.400683       1 flags.go:64] FLAG: --tls-private-key-file="/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key"
I0131 07:32:09.400687       1 flags.go:64] FLAG: --tls-sni-cert-key="[]"
I0131 07:32:09.400693       1 flags.go:64] FLAG: --unhealthy-zone-threshold="0.55"
I0131 07:32:09.400696       1 flags.go:64] FLAG: --unsupported-kube-api-over-localhost="false"
I0131 07:32:09.400699       1 flags.go:64] FLAG: --use-service-account-credentials="true"
I0131 07:32:09.400703       1 flags.go:64] FLAG: --v="2"
I0131 07:32:09.400707       1 flags.go:64] FLAG: --version="false"
I0131 07:32:09.400718       1 flags.go:64] FLAG: --vmodule=""
I0131 07:32:09.400722       1 flags.go:64] FLAG: --volume-host-allow-local-loopback="true"
I0131 07:32:09.400725       1 flags.go:64] FLAG: --volume-host-cidr-denylist="[]"
I0131 07:32:09.402141       1 dynamic_serving_content.go:112] "Loaded a new cert/key pair" name="serving-cert::/et

[knarra@knarra cucushift]$ oc logs kube-controller-manager-ip-10-0-192-9.us-east-2.compute.internal -n openshift-kube-controller-manager | grep "unrecognized feature gate"
[knarra@knarra cucushift]$ 

[knarra@knarra cucushift]$ oc logs kube-controller-manager-ip-10-0-192-9.us-east-2.compute.internal -n openshift-kube-controller-manager | grep "unrecognized"
[knarra@knarra cucushift]$ 



kube-apiserver logs:
====================
I0131 07:27:42.172438      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true]}
I0131 07:27:42.172504      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true RotateKubeletServerCertificate:true]}
I0131 07:27:42.172543      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true DownwardAPIHugePages:true RotateKubeletServerCertificate:true]}
I0131 07:27:42.172599      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0131 07:27:42.172640      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0131 07:27:42.172675      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false CSIMigrationOpenStack:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServe
rCertificate:true]}
I0131 07:27:42.172714      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false CSIMigrationGCE:false CSIMigrationOpenStack:false DownwardAPIHugePages:true PodSecurity:t
rue RotateKubeletServerCertificate:true]}
I0131 07:27:42.172751      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false CSIMigrationAzureDisk:false CSIMigrationGCE:false CSIMigrationOpenStack:false DownwardAPI
HugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0131 07:27:42.172847      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false CSIMigrationAzureDisk:false CSIMigrationAzureFile:false CSIMigrationGCE:false CSIMigratio
nOpenStack:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0131 07:27:42.172888      16 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false CSIMigrationAzureDisk:false CSIMigrationAzureFile:false CSIMigrationGCE:false CSIMigratio
nOpenStack:false CSIMigrationvSphere:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}

[knarra@knarra cucushift]$ oc logs kube-apiserver-ip-10-0-178-236.us-east-2.compute.internal -n openshift-kube-apiserver | grep "unrecognized feature gate"
[knarra@knarra cucushift]$ 

[knarra@knarra cucushift]$ oc logs kube-apiserver-ip-10-0-178-236.us-east-2.compute.internal -n openshift-kube-apiserver | grep "unrecognized"
[knarra@knarra cucushift]$ 


Based on the above moving bug to verified state.

Comment 12 errata-xmlrpc 2022-03-12 04:41:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.