Bug 1960205
| Summary: | The kubelet log flooded with reconcileState message once CPU manager enabled | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Artyom <alukiano> |
| Component: | Node | Assignee: | Jiří Mencák <jmencak> |
| Node sub component: | CPU manager | QA Contact: | Walid A. <wabouham> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | aos-bugs, rphillips |
| Version: | 4.8 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.8.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-27 23:08:23 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Artyom
2021-05-13 10:24:51 UTC
*** Bug 1960206 has been marked as a duplicate of this bug. *** On OCP 4.8 IPI cluster on AWS, 4.8.0-0.nightly-2021-06-01-043518, I am still seeing several thousands of messages with CPU manager enabled: sh-4.4# sh-4.4# journalctl --no-pager --since yesterday | grep -c "cpu_manager" 33263 sh-4.4# sh-4.4# journalctl --no-pager --since yesterday | grep -c "cpu_manager.go:412" 33242 sh-4.4# sh-4.4# journalctl --no-pager | grep -c "cpu_manager.go:412" 33260 sh-4.4# sh-4.4# journalctl --no-pager | grep -c "cpu_manager.go:412] \"ReconcileState: ignoring terminated container" 33326 sh-4.4# sh-4.4# journalctl --no-pager -f -- Logs begin at Tue 2021-06-01 13:59:02 UTC. -- Jun 02 04:27:52 ip-10-0-149-94 hyperkube[1395]: I0602 04:27:52.328126 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="aa63191d5ebbdd39d28e28f42813723663b0fbfc45705664e94bf973d678ed65" Jun 02 04:27:52 ip-10-0-149-94 hyperkube[1395]: I0602 04:27:52.328139 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="349ffa6e1d82d090d954b5b63a68996cacf21e75f7f39eff75a0455ddd698b1f" Jun 02 04:27:52 ip-10-0-149-94 hyperkube[1395]: I0602 04:27:52.328152 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="8ed635204e177dc16bef900dae11ebbba332615a47b182e13785810cfb1dade3" Jun 02 04:27:52 ip-10-0-149-94 hyperkube[1395]: I0602 04:27:52.328167 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="227fc478587b72b1531cd90eb9e20c9f87d87a403b824f37f6be983a303e0d82" Jun 02 04:27:57 ip-10-0-149-94 hyperkube[1395]: I0602 04:27:57.857372 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-monitoring/node-exporter-95llb" containerID="c1e1f6232872768f0fec6677a3910d912ef3d81903cfd0302d0ad029b45fd94e" Jun 02 04:27:58 ip-10-0-149-94 hyperkube[1395]: I0602 04:27:58.208880 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="783c24dcb86857d3e7a2ad9c5edf32cc567919e40c7465e687eeff48df3ddb0d" Jun 02 04:27:58 ip-10-0-149-94 hyperkube[1395]: I0602 04:27:58.208912 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="aa63191d5ebbdd39d28e28f42813723663b0fbfc45705664e94bf973d678ed65" Jun 02 04:27:58 ip-10-0-149-94 hyperkube[1395]: I0602 04:27:58.208931 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="349ffa6e1d82d090d954b5b63a68996cacf21e75f7f39eff75a0455ddd698b1f" Jun 02 04:27:58 ip-10-0-149-94 hyperkube[1395]: I0602 04:27:58.208946 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="8ed635204e177dc16bef900dae11ebbba332615a47b182e13785810cfb1dade3" Jun 02 04:27:58 ip-10-0-149-94 hyperkube[1395]: I0602 04:27:58.208961 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="227fc478587b72b1531cd90eb9e20c9f87d87a403b824f37f6be983a303e0d82" Jun 02 04:28:03 ip-10-0-149-94 hyperkube[1395]: I0602 04:28:03.676608 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-monitoring/node-exporter-95llb" containerID="c1e1f6232872768f0fec6677a3910d912ef3d81903cfd0302d0ad029b45fd94e" Jun 02 04:28:04 ip-10-0-149-94 hyperkube[1395]: I0602 04:28:04.061887 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="783c24dcb86857d3e7a2ad9c5edf32cc567919e40c7465e687eeff48df3ddb0d" Jun 02 04:28:04 ip-10-0-149-94 hyperkube[1395]: I0602 04:28:04.061917 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="aa63191d5ebbdd39d28e28f42813723663b0fbfc45705664e94bf973d678ed65" Jun 02 04:28:04 ip-10-0-149-94 hyperkube[1395]: I0602 04:28:04.061931 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="349ffa6e1d82d090d954b5b63a68996cacf21e75f7f39eff75a0455ddd698b1f" Jun 02 04:28:04 ip-10-0-149-94 hyperkube[1395]: I0602 04:28:04.061946 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="8ed635204e177dc16bef900dae11ebbba332615a47b182e13785810cfb1dade3" Jun 02 04:28:04 ip-10-0-149-94 hyperkube[1395]: I0602 04:28:04.061959 1395 cpu_manager.go:412] "ReconcileState: ignoring terminated container" pod="openshift-multus/multus-additional-cni-plugins-sgbgb" containerID="227fc478587b72b1531cd90eb9e20c9f87d87a403b824f37f6be983a303e0d82" ^C sh-4.4# Thanks for looking into this, Walid. This is expected and I also noticed and discussed this particular log message with Artyom. For the moment we decided to address only the '[cpumanager] reconcileState: skipping pod; pod is managed ' message, introduced in the OpenShift-specific (openshift/kubernetes) repo by https://github.com/openshift/kubernetes/pull/627 The messages you are seeing have not been introduced by PR#627 and could potentially be addressed in the upstream kubernetes/kubernetes repo. However, this particular BZ report is not about them. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |