Bug 1976379
| Summary: | CVO pod skipped by workload partitioning with incorrect error stating cluster is not SNO | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Ian Miller <imiller> | ||||
| Component: | Node | Assignee: | Artyom <alukiano> | ||||
| Node sub component: | Autoscaler (HPA, VPA) | QA Contact: | Sunil Choudhary <schoudha> | ||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||
| Severity: | high | ||||||
| Priority: | high | CC: | aos-bugs, browsell, dhellmann, kewang, keyoung, mfojtik, sttts, wking, xxia | ||||
| Version: | 4.8 | Flags: | imiller:
needinfo-
|
||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.9.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-10-18 17:36:53 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1977351 | ||||||
| Attachments: |
|
||||||
Can you please provide the installer debug log? First tried to study https://github.com/openshift/enhancements/blob/master/enhancements/workload-partitioning/management-workload-partitioning.md to get the background (due to it is Node scope, I'm moving to Node component Autoscaler subcomponent but I'll continue to verify it). Second, tried in old unfixed version like 4.8.0-0.nightly-2021-07-02-143616 in non-SNO env: oc get po -n openshift-cluster-version -o yaml # can easily see the incorrect warning: metadata: annotations: workload.openshift.io/warning: only single-node clusters support workload partitioning name: cluster-version-operator-78f9799669-knfsz oc describe no POD_NODE | grep cluster-version-operator # can easily see 20m CPU request for the CVO pod Then tried latest 4.9 non-SNO env (4.9.0-0.nightly-2021-07-04-140102 as of now), the issue still exists. Then checked 4.9.0-0.nightly-2021-07-04-140102, its last o/k commit is 0228142e087: oc adm release info --commits registry.ci.openshift.org/ocp/release:4.9.0-0.nightly-2021-07-04-140102 | grep hyperkube hyperkube https://github.com/openshift/kubernetes 0228142e087... https://github.com/openshift/kubernetes/blob/0228142e087/openshift-kube-apiserver/admission/autoscaling/managementcpusoverride/admission.go#L183-L186 already contains the fix of PR. Thus moving back to ASSIGNED I think the title of the bug misleads you, the bug should be verified on the SNO with the workload partitioning enabled. Can you please verify on the SNO environment with the workload partition enabled? (In reply to Artyom from comment #4) Thanks for clarification. Then it is better to ask help from the QE colleague from team of the workload partitioning feature. Let me update. Checked on SNO setup with workload partitioning configured during cluster deployment.
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.9.0-rc.0 True False 3h8m Cluster version is 4.9.0-rc.0
$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master-0.cluster1.savanna.lab.eng.rdu2.redhat.com Ready master,worker 3h26m v1.22.0-rc.0+75ee307 10.1.190.12 <none> Red Hat Enterprise Linux CoreOS 49.84.202109041651-0 (Ootpa) 4.18.0-305.17.1.rt7.89.el8_4.x86_64 cri-o://1.22.0-68.rhaos4.9.git011c10a.el8
$ oc describe mc 02-master-workload-partitioning
Name: 02-master-workload-partitioning
Namespace:
Labels: machineconfiguration.openshift.io/role=master
Annotations: <none>
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Metadata:
Creation Timestamp: 2021-09-14T12:55:06Z
Generation: 1
Managed Fields:
API Version: machineconfiguration.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:machineconfiguration.openshift.io/role:
f:spec:
.:
f:config:
.:
f:ignition:
.:
f:version:
f:storage:
.:
f:files:
Manager: cluster-bootstrap
Operation: Update
Time: 2021-09-14T12:55:06Z
Resource Version: 1874
UID: 330a580d-b29a-4fa6-b872-7add31619671
Spec:
Config:
Ignition:
Version: 3.2.0
Storage:
Files:
Contents:
Source: data:text/plain;charset=utf-8;base64,W2NyaW8ucnVudGltZS53b3JrbG9hZHMubWFuYWdlbWVudF0KYWN0aXZhdGlvbl9hbm5vdGF0aW9uID0gInRhcmdldC53b3JrbG9hZC5vcGVuc2hpZnQuaW8vbWFuYWdlbWVudCIKYW5ub3RhdGlvbl9wcmVmaXggPSAicmVzb3VyY2VzLndvcmtsb2FkLm9wZW5zaGlmdC5pbyIKcmVzb3VyY2VzID0geyAiY3B1c2hhcmVzIiA9IDAsICJjcHVzZXQiID0gIjAtMSw0MC00MSIgfQo=
Mode: 420
Overwrite: true
Path: /etc/crio/crio.conf.d/01-workload-partitioning
User:
Name: root
Contents:
Source: data:text/plain;charset=utf-8;base64,ewogICJtYW5hZ2VtZW50IjogewogICAgImNwdXNldCI6ICIwLTEsNDAtNDEiCiAgfQp9Cg==
Mode: 420
Overwrite: true
Path: /etc/kubernetes/openshift-workload-pinning
User:
Name: root
Events: <none>
$ oc describe node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com
Name: master-0.cluster1.savanna.lab.eng.rdu2.redhat.com
Roles: master,worker
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=master-0.cluster1.savanna.lab.eng.rdu2.redhat.com
kubernetes.io/os=linux
node-role.kubernetes.io/master=
node-role.kubernetes.io/worker=
node.openshift.io/os_id=rhcos
Annotations: k8s.ovn.org/host-addresses: ["10.1.190.12","10.1.190.40","10.1.190.44","172.22.0.166"]
k8s.ovn.org/l3-gateway-config:
{"default":{"mode":"shared","interface-id":"br-ex_master-0.cluster1.savanna.lab.eng.rdu2.redhat.com","mac-address":"98:03:9b:8e:86:c4","ip...
k8s.ovn.org/node-chassis-id: a7ddf241-881e-45f1-8809-8541528be673
k8s.ovn.org/node-mgmt-port-mac-address: 56:ce:0e:59:6f:09
k8s.ovn.org/node-primary-ifaddr: {"ipv4":"10.1.190.12/23"}
k8s.ovn.org/node-subnets: {"default":"10.128.0.0/23"}
k8s.ovn.org/topology-version: 4
machineconfiguration.openshift.io/controlPlaneTopology: SingleReplica
machineconfiguration.openshift.io/currentConfig: rendered-master-c61924ab4910008fe9f7d47ea1d19dad
machineconfiguration.openshift.io/desiredConfig: rendered-master-c61924ab4910008fe9f7d47ea1d19dad
machineconfiguration.openshift.io/reason:
machineconfiguration.openshift.io/ssh: accessed
machineconfiguration.openshift.io/state: Done
sriovnetwork.openshift.io/state: Idle
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 14 Sep 2021 18:40:24 +0530
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: master-0.cluster1.savanna.lab.eng.rdu2.redhat.com
AcquireTime: <unset>
RenewTime: Tue, 14 Sep 2021 22:07:34 +0530
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 14 Sep 2021 22:02:36 +0530 Tue, 14 Sep 2021 18:40:24 +0530 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 14 Sep 2021 22:02:36 +0530 Tue, 14 Sep 2021 18:40:24 +0530 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 14 Sep 2021 22:02:36 +0530 Tue, 14 Sep 2021 18:40:24 +0530 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 14 Sep 2021 22:02:36 +0530 Tue, 14 Sep 2021 18:46:10 +0530 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.1.190.12
Hostname: master-0.cluster1.savanna.lab.eng.rdu2.redhat.com
Capacity:
cpu: 80
ephemeral-storage: 1561964428Ki
hugepages-1Gi: 32Gi
hugepages-2Mi: 0
management.workload.openshift.io/cores: 80k
memory: 394645704Ki
openshift.io/du_mh: 8
pods: 250
Allocatable:
cpu: 76
ephemeral-storage: 1561964428Ki
hugepages-1Gi: 32Gi
hugepages-2Mi: 0
management.workload.openshift.io/cores: 80k
memory: 359964872Ki
openshift.io/du_mh: 8
pods: 250
System Info:
Machine ID: 8869ab0904fb429f9e338d90028ff69c
System UUID: 4c4c4544-0048-3510-8051-b3c04f505832
Boot ID: e682eec2-a8f3-410a-92dc-8a424477f3c8
Kernel Version: 4.18.0-305.17.1.rt7.89.el8_4.x86_64
OS Image: Red Hat Enterprise Linux CoreOS 49.84.202109041651-0 (Ootpa)
Operating System: linux
Architecture: amd64
Container Runtime Version: cri-o://1.22.0-68.rhaos4.9.git011c10a.el8
Kubelet Version: v1.22.0-rc.0+75ee307
Kube-Proxy Version: v1.22.0-rc.0+75ee307
Non-terminated Pods: (96 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
open-cluster-management-agent-addon klusterlet-addon-appmgr-68c6c47fc6-dh6p7 0 (0%) 0 (0%) 128Mi (0%) 2Gi (0%) 3h1m
open-cluster-management-agent-addon klusterlet-addon-operator-6f557446b6-v8xmw 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h4m
open-cluster-management-agent-addon klusterlet-addon-policyctrl-config-policy-779cd698bb-db9wt 0 (0%) 0 (0%) 128Mi (0%) 256Mi (0%) 3h1m
open-cluster-management-agent-addon klusterlet-addon-policyctrl-framework-6879ffdf9b-jmscq 0 (0%) 0 (0%) 384Mi (0%) 768Mi (0%) 3h1m
open-cluster-management-agent-addon klusterlet-addon-workmgr-5d8d97bcd5-2l8kq 0 (0%) 0 (0%) 128Mi (0%) 512Mi (0%) 3h1m
open-cluster-management-agent klusterlet-7dbdb54759-mfvcg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h6m
open-cluster-management-agent klusterlet-registration-agent-6bd475b87c-gqdrt 0 (0%) 0 (0%) 128Mi (0%) 0 (0%) 3h5m
open-cluster-management-agent klusterlet-work-agent-6bfdcc9ccf-vbsm8 0 (0%) 0 (0%) 128Mi (0%) 0 (0%) 3h5m
openshift-apiserver-operator openshift-apiserver-operator-67475f6559-7p6x8 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m
openshift-apiserver apiserver-7584f94579-99j9d 0 (0%) 0 (0%) 250Mi (0%) 0 (0%) 3h14m
openshift-authentication-operator authentication-operator-697f847cb-6n7mz 0 (0%) 0 (0%) 200Mi (0%) 0 (0%) 3h26m
openshift-authentication oauth-openshift-855d946b6b-dpptv 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h11m
openshift-cloud-controller-manager-operator cluster-cloud-controller-manager-operator-6c45546cfb-kzhmk 0 (0%) 0 (0%) 75Mi (0%) 0 (0%) 3h15m
openshift-cloud-credential-operator cloud-credential-operator-765479c5d8-8cmsh 0 (0%) 0 (0%) 170Mi (0%) 0 (0%) 3h26m
openshift-cluster-machine-approver machine-approver-5bc975c7f7-lx997 0 (0%) 0 (0%) 70Mi (0%) 0 (0%) 3h26m
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-6544b7f674-c8x6f 0 (0%) 0 (0%) 20Mi (0%) 0 (0%) 3h26m
openshift-cluster-node-tuning-operator tuned-lk7m2 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h21m
openshift-cluster-samples-operator cluster-samples-operator-754cdd9744-m5982 0 (0%) 0 (0%) 100Mi (0%) 0 (0%) 3h15m
openshift-cluster-storage-operator cluster-storage-operator-6554f956d8-tdgsh 0 (0%) 0 (0%) 20Mi (0%) 0 (0%) 3h15m
openshift-cluster-storage-operator csi-snapshot-controller-97dfb65b7-x5ggx 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h21m
openshift-cluster-storage-operator csi-snapshot-controller-operator-7b87d997b7-gk7f2 0 (0%) 0 (0%) 65Mi (0%) 0 (0%) 3h26m
openshift-cluster-storage-operator csi-snapshot-webhook-987b4bc98-t5fd7 0 (0%) 0 (0%) 20Mi (0%) 0 (0%) 3h21m
openshift-cluster-version cluster-version-operator-7cd8b7f756-5vrs5 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m
openshift-config-operator openshift-config-operator-6d5b45f545-sh98v 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m
openshift-console-operator console-operator-6b6f56c57f-bwrhx 0 (0%) 0 (0%) 100Mi (0%) 0 (0%) 3h15m
openshift-controller-manager-operator openshift-controller-manager-operator-5cdb6958f5-6m4vh 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m
openshift-controller-manager controller-manager-xm5p2 0 (0%) 0 (0%) 100Mi (0%) 0 (0%) 3h15m
openshift-dns-operator dns-operator-67968cf9f9-vhgtr 0 (0%) 0 (0%) 69Mi (0%) 0 (0%) 3h26m
openshift-dns dns-default-c9vkf 0 (0%) 0 (0%) 110Mi (0%) 0 (0%) 3h22m
openshift-dns node-resolver-5f4qv 0 (0%) 0 (0%) 21Mi (0%) 0 (0%) 3h22m
openshift-etcd-operator etcd-operator-88785ddc-g55cb 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m
openshift-etcd etcd-master-0.cluster1.savanna.lab.eng.rdu2.redhat.com 0 (0%) 0 (0%) 930Mi (0%) 0 (0%) 3h17m
openshift-image-registry cluster-image-registry-operator-67586bb65b-7jzlf 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h15m
openshift-image-registry node-ca-56h5b 0 (0%) 0 (0%) 10Mi (0%) 0 (0%) 3h12m
openshift-ingress-canary ingress-canary-qrjtk 0 (0%) 0 (0%) 20Mi (0%) 0 (0%) 3h12m
openshift-ingress-operator ingress-operator-9fcf4fcbf-g8fwr 0 (0%) 0 (0%) 96Mi (0%) 0 (0%) 3h15m
openshift-ingress router-default-5757cfd87-d8g9g 0 (0%) 0 (0%) 256Mi (0%) 0 (0%) 3h12m
openshift-insights insights-operator-5f8b67548b-wnzpl 0 (0%) 0 (0%) 30Mi (0%) 0 (0%) 3h15m
openshift-kube-apiserver-operator kube-apiserver-operator-565b5d9bbc-t7gjw 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m
openshift-kube-apiserver kube-apiserver-master-0.cluster1.savanna.lab.eng.rdu2.redhat.com 0 (0%) 0 (0%) 1224Mi (0%) 0 (0%) 3h10m
openshift-kube-controller-manager-operator kube-controller-manager-operator-86bb8c68fb-mljnl 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m
openshift-kube-controller-manager kube-controller-manager-master-0.cluster1.savanna.lab.eng.rdu2.redhat.com 0 (0%) 0 (0%) 500Mi (0%) 0 (0%) 3h11m
openshift-kube-scheduler-operator openshift-kube-scheduler-operator-f5c467697-cvkb9 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m
openshift-kube-scheduler openshift-kube-scheduler-master-0.cluster1.savanna.lab.eng.rdu2.redhat.com 0 (0%) 0 (0%) 150Mi (0%) 0 (0%) 3h11m
openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-5d69c9c8bf-h6gzd 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m
openshift-kube-storage-version-migrator migrator-798cb9b757-bvlbn 0 (0%) 0 (0%) 200Mi (0%) 0 (0%) 3h22m
openshift-local-storage local-storage-operator-7f6477ddfd-fhxm4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 166m
openshift-logging cluster-logging-operator-5df48cc674-qprsj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 166m
openshift-logging fluentd-9sl7x 0 (0%) 0 (0%) 736Mi (0%) 736Mi (0%) 150m
openshift-machine-api cluster-autoscaler-operator-766bb4796f-wpdzx 0 (0%) 0 (0%) 70Mi (0%) 0 (0%) 3h15m
openshift-machine-api cluster-baremetal-operator-79f5f7c994-6jpdc 0 (0%) 0 (0%) 70Mi (0%) 0 (0%) 3h15m
openshift-machine-api machine-api-operator-bd8ffc766-659fw 0 (0%) 0 (0%) 70Mi (0%) 0 (0%) 3h15m
openshift-machine-config-operator machine-config-controller-655f487b68-jlcpn 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h21m
openshift-machine-config-operator machine-config-daemon-gqrhc 0 (0%) 0 (0%) 100Mi (0%) 0 (0%) 3h22m
openshift-machine-config-operator machine-config-operator-78c44d9f9c-znwcv 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m
openshift-machine-config-operator machine-config-server-7pz4p 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h21m
openshift-marketplace cluster-logging-operator-brqfl 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 175m
openshift-marketplace local-storage-operator-48sw9 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h
openshift-marketplace marketplace-operator-6d49688f8c-x8cfr 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m
openshift-marketplace performance-addon-operator-9trqk 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 175m
openshift-marketplace ptp-operator-xvqr9 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 174m
openshift-marketplace sriov-network-operator-r5bh6 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h
openshift-monitoring alertmanager-main-0 0 (0%) 0 (0%) 105Mi (0%) 0 (0%) 3h12m
openshift-monitoring cluster-monitoring-operator-85f6566b9-84hvz 0 (0%) 0 (0%) 95Mi (0%) 0 (0%) 3h26m
openshift-monitoring grafana-5bf796858d-7dd6d 0 (0%) 0 (0%) 84Mi (0%) 0 (0%) 3h12m
openshift-monitoring kube-state-metrics-7c7fb896db-5k879 0 (0%) 0 (0%) 110Mi (0%) 0 (0%) 3h21m
openshift-monitoring node-exporter-d5c2c 0 (0%) 0 (0%) 47Mi (0%) 0 (0%) 3h21m
openshift-monitoring openshift-state-metrics-69b88f859f-mhlg8 0 (0%) 0 (0%) 72Mi (0%) 0 (0%) 3h21m
openshift-monitoring prometheus-adapter-5fb75fd796-bgspj 0 (0%) 0 (0%) 40Mi (0%) 0 (0%) 3h15m
openshift-monitoring prometheus-k8s-0 0 (0%) 0 (0%) 1119Mi (0%) 0 (0%) 3h12m
openshift-monitoring prometheus-operator-7c8d859ddc-tsrbd 0 (0%) 0 (0%) 165Mi (0%) 0 (0%) 3h22m
openshift-monitoring thanos-querier-5779d46758-c5vhg 0 (0%) 0 (0%) 77Mi (0%) 0 (0%) 3h12m
openshift-multus multus-4nd96 0 (0%) 0 (0%) 65Mi (0%) 0 (0%) 3h24m
openshift-multus multus-additional-cni-plugins-tfms6 0 (0%) 0 (0%) 10Mi (0%) 0 (0%) 3h24m
openshift-multus multus-admission-controller-99f8p 0 (0%) 0 (0%) 70Mi (0%) 0 (0%) 3h22m
openshift-multus network-metrics-daemon-pqhp8 0 (0%) 0 (0%) 120Mi (0%) 0 (0%) 3h24m
openshift-network-diagnostics network-check-source-6b4b6f7445-k5zb8 0 (0%) 0 (0%) 40Mi (0%) 0 (0%) 3h24m
openshift-network-diagnostics network-check-target-qqnkg 0 (0%) 0 (0%) 15Mi (0%) 0 (0%) 3h24m
openshift-network-operator network-operator-5b9cd89648-qk9bx 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m
openshift-oauth-apiserver apiserver-56d7684bb6-l7gs2 0 (0%) 0 (0%) 200Mi (0%) 0 (0%) 3h22m
openshift-operator-lifecycle-manager catalog-operator-764dd45b5d-674d9 0 (0%) 0 (0%) 80Mi (0%) 0 (0%) 3h15m
openshift-operator-lifecycle-manager olm-operator-794d77cdb6-krnjd 0 (0%) 0 (0%) 160Mi (0%) 0 (0%) 3h15m
openshift-operator-lifecycle-manager package-server-manager-fcd7dc6d-wrw49 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h15m
openshift-operator-lifecycle-manager packageserver-b8f685656-68fnl 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h13m
openshift-ovn-kubernetes ovnkube-master-swtz9 0 (0%) 0 (0%) 1520Mi (0%) 0 (0%) 3h24m
openshift-ovn-kubernetes ovnkube-node-clj9p 0 (0%) 0 (0%) 640Mi (0%) 0 (0%) 3h24m
openshift-performance-addon-operator performance-operator-556c5b77db-8gtng 0 (0%) 0 (0%) 25Mi (0%) 0 (0%) 165m
openshift-ptp linuxptp-daemon-sxtkg 0 (0%) 0 (0%) 70Mi (0%) 0 (0%) 125m
openshift-ptp ptp-operator-54c64b6c5f-zwvc7 0 (0%) 0 (0%) 100Mi (0%) 0 (0%) 126m
openshift-service-ca-operator service-ca-operator-5b7bd8c945-tv8rx 0 (0%) 0 (0%) 80Mi (0%) 0 (0%) 3h26m
openshift-service-ca service-ca-64cf77d8f8-48r4s 0 (0%) 0 (0%) 120Mi (0%) 0 (0%) 3h22m
openshift-sriov-network-operator network-resources-injector-hsc77 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 125m
openshift-sriov-network-operator operator-webhook-mdnlc 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 125m
openshift-sriov-network-operator sriov-device-plugin-x6gmg 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 112m
openshift-sriov-network-operator sriov-network-config-daemon-25v6k 0 (0%) 0 (0%) 120Mi (0%) 0 (0%) 125m
openshift-sriov-network-operator sriov-network-operator-869cbff7bc-hz92p 0 (0%) 0 (0%) 0 (0%) 0 (0%) 126m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 0 (0%) 0 (0%)
memory 13625Mi (3%) 4320Mi (1%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
management.workload.openshift.io/cores 2871 2871
openshift.io/du_mh 0 0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeReady 3h22m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeReady
Normal NodeAllocatableEnforced 3h22m kubelet Updated Node Allocatable limit across pods
Normal Starting 3h22m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 3h22m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 3h22m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure
Normal NodeNotReady 3h22m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeNotReady
Normal NodeHasSufficientPID 3h22m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID
Normal NodeReady 3h22m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeReady
Normal NodeNotReady 3h21m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeNotReady
Normal Starting 3h21m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 3h21m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory
Normal NodeHasSufficientPID 3h21m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID
Normal NodeHasNoDiskPressure 3h21m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure
Normal NodeAllocatableEnforced 3h21m kubelet Updated Node Allocatable limit across pods
Normal NodeReady 3h21m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeReady
Normal Starting 177m kubelet Starting kubelet.
Normal NodeAllocatableEnforced 177m kubelet Updated Node Allocatable limit across pods
Normal NodeHasSufficientMemory 177m (x8 over 177m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 177m (x8 over 177m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 177m (x7 over 177m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 170m kubelet Updated Node Allocatable limit across pods
Normal NodeHasSufficientMemory 170m (x8 over 170m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 170m (x8 over 170m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 170m (x7 over 170m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID
Normal Starting 170m kubelet Starting kubelet.
Normal Starting 161m kubelet Starting kubelet.
Normal NodeHasNoDiskPressure 161m (x8 over 161m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 161m (x7 over 161m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 161m kubelet Updated Node Allocatable limit across pods
Normal NodeHasSufficientMemory 161m (x8 over 161m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory
Normal NodeAllocatableEnforced 153m kubelet Updated Node Allocatable limit across pods
Normal Starting 153m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 153m (x8 over 153m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory
Normal NodeHasSufficientPID 153m (x7 over 153m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID
Normal NodeHasNoDiskPressure 153m (x8 over 153m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure
Normal Starting 116m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 116m (x8 over 116m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 116m (x8 over 116m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 116m (x7 over 116m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 116m kubelet Updated Node Allocatable limit across pods
$ oc describe node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com | grep -i openshift-cluster-version
openshift-cluster-version cluster-version-operator-7cd8b7f756-5vrs5 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h27m
$ oc describe pod cluster-version-operator-7cd8b7f756-5vrs5 -n openshift-cluster-version
Name: cluster-version-operator-7cd8b7f756-5vrs5
Namespace: openshift-cluster-version
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: master-0.cluster1.savanna.lab.eng.rdu2.redhat.com/10.1.190.12
Start Time: Tue, 14 Sep 2021 18:41:03 +0530
Labels: k8s-app=cluster-version-operator
pod-template-hash=7cd8b7f756
Annotations: resources.workload.openshift.io/cluster-version-operator: {"cpushares": 20}
target.workload.openshift.io/management: {"effect":"PreferredDuringScheduling"}
Status: Running
IP: 10.1.190.12
IPs:
IP: 10.1.190.12
Controlled By: ReplicaSet/cluster-version-operator-7cd8b7f756
Containers:
cluster-version-operator:
Container ID: cri-o://0f7982f0f4c90a16f5df9d16ebf50ed3e3ebd58b9c08eb1a352d4b02d4a3e19e
Image: registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca
Image ID: registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca
Port: <none>
Host Port: <none>
Args:
start
--release-image=registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca
--enable-auto-update=false
--enable-default-cluster-version=true
--listen=0.0.0.0:9099
--serving-cert-file=/etc/tls/serving-cert/tls.crt
--serving-key-file=/etc/tls/serving-cert/tls.key
--v=5
State: Running
Started: Tue, 14 Sep 2021 20:11:39 +0530
Ready: True
Restart Count: 5
Limits:
management.workload.openshift.io/cores: 20
Requests:
management.workload.openshift.io/cores: 20
memory: 50Mi
Environment:
KUBERNETES_SERVICE_PORT: 6443
KUBERNETES_SERVICE_HOST: 127.0.0.1
NODE_NAME: (v1:spec.nodeName)
CLUSTER_PROFILE: self-managed-high-availability
Mounts:
/etc/cvo/updatepayloads from etc-cvo-updatepayloads (ro)
/etc/ssl/certs from etc-ssl-certs (ro)
/etc/tls/serving-cert from serving-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wv2cn (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
etc-ssl-certs:
Type: HostPath (bare host directory volume)
Path: /etc/ssl/certs
HostPathType:
etc-cvo-updatepayloads:
Type: HostPath (bare host directory volume)
Path: /etc/cvo/updatepayloads
HostPathType:
serving-cert:
Type: Secret (a volume populated by a Secret)
SecretName: cluster-version-operator-serving-cert
Optional: false
kube-api-access-wv2cn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
ConfigMapName: openshift-service-ca.crt
ConfigMapOptional: <nil>
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/master=
Tolerations: node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 120s
node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3h28m default-scheduler Successfully assigned openshift-cluster-version/cluster-version-operator-7cd8b7f756-5vrs5 to master-0.cluster1.savanna.lab.eng.rdu2.redhat.com
Warning FailedMount 3h26m kubelet Unable to attach or mount volumes: unmounted volumes=[serving-cert], unattached volumes=[kube-api-access-wv2cn etc-ssl-certs etc-cvo-updatepayloads serving-cert]: timed out waiting for the condition
Warning FailedMount 3h26m (x9 over 3h28m) kubelet MountVolume.SetUp failed for volume "serving-cert" : secret "cluster-version-operator-serving-cert" not found
Normal Pulling 3h24m kubelet Pulling image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca"
Normal Pulled 3h24m kubelet Successfully pulled image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" in 3.194995787s
Normal Created 3h24m kubelet Created container cluster-version-operator
Normal Started 3h24m kubelet Started container cluster-version-operator
Normal Pulled 178m kubelet Container image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" already present on machine
Normal Created 178m kubelet Created container cluster-version-operator
Normal Started 178m kubelet Started container cluster-version-operator
Warning Failed 172m kubelet Error: services have not yet been read at least once, cannot construct envvars
Normal Pulled 172m (x2 over 172m) kubelet Container image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" already present on machine
Normal Created 172m kubelet Created container cluster-version-operator
Normal Started 172m kubelet Started container cluster-version-operator
Normal Pulled 163m kubelet Container image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" already present on machine
Normal Created 163m kubelet Created container cluster-version-operator
Normal Started 163m kubelet Started container cluster-version-operator
Normal Pulled 155m kubelet Container image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" already present on machine
Normal Created 155m kubelet Created container cluster-version-operator
Normal Started 155m kubelet Started container cluster-version-operator
Normal Pulled 117m kubelet Container image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" already present on machine
Normal Created 117m kubelet Created container cluster-version-operator
Normal Started 117m kubelet Started container cluster-version-operator
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759 |
Created attachment 1794553 [details] must-gather from cluster where this occurred Description of problem: Pod "cluster-version-operator-89bf5cdb5-4qhhh" in openshift-cluster-version namespace was not handled by the workload partitioning pod mutation logic. A warning was added to the pod: apiVersion: v1 kind: Pod metadata: annotations: workload.openshift.io/warning: only single-node clusters support workload partitioning Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-06-24-222938 How reproducible: unknown Steps to Reproduce: 1. Cluster installed 2. "oc describe node" shows 20m CPU requests for this pod 3. Actual results: openshift-cluster-version cluster-version-operator-89bf5cdb5-4qhhh 20m (0%) 0 (0%) 50Mi (0%) 0 (0%) 4h Expected results: openshift-cluster-version cluster-version-operator-89bf5cdb5-4qhhh 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 4h Additional info: