Description of problem: When using a custom pool for a subset of the worker nodes, the worker MCP shows an outdated machine count until the MCP is synced again (this can also be manually triggered by a worker MCP update like setting maxUnavailable). Version-Release number of selected component (if applicable): 4.2 How reproducible: 100% Steps to Reproduce: 1. set a node to be infra and create an infra MCP/MC inheriting from worker 2. oc describe mcp worker 3. Actual results: On a 3 nodes cluster and 1 infra node: oc describe mcp worker ... Machine Count: 3 ... Expected results: oc describe mcp worker ... Machine Count: 2 ... Additional info:
Verified on 4.2.0-0.nightly-2019-08-20-162755 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.2.0-0.nightly-2019-08-20-162755 True False 11m Cluster version is 4.2.0-0.nightly-2019-08-20-162755 cat mcpinfra.yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfigPool metadata: name: infra spec: machineConfigSelector: matchExpressions: - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]} maxUnavailable: null nodeSelector: matchLabels: node-role.kubernetes.io/infra: "" paused: false $ oc create -f mcpinfra.yaml machineconfigpool.machineconfiguration.openshift.io/infra created $ oc get machineconfigpool NAME CONFIG UPDATED UPDATING DEGRADED infra rendered-infra-0fbb7c0ce09edd8e04bf68881c95d62c True False False master rendered-master-72b9921bcfbd8242bfb984850defee68 True False False worker rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c True False False $ oc get machineconfigpool worker -o yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfigPool metadata: creationTimestamp: "2019-08-21T14:26:46Z" generation: 2 labels: machineconfiguration.openshift.io/mco-built-in: "" name: worker resourceVersion: "13234" selfLink: /apis/machineconfiguration.openshift.io/v1/machineconfigpools/worker uid: b46bbcb7-c41f-11e9-bb1a-0239f61aa074 spec: configuration: name: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c source: - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 00-worker - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 01-worker-container-runtime - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 01-worker-kubelet - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 99-worker-ssh machineConfigSelector: matchLabels: machineconfiguration.openshift.io/role: worker nodeSelector: matchLabels: node-role.kubernetes.io/worker: "" paused: false status: conditions: - lastTransitionTime: "2019-08-21T14:27:12Z" message: "" reason: "" status: "False" type: RenderDegraded - lastTransitionTime: "2019-08-21T14:27:15Z" message: "" reason: "" status: "False" type: NodeDegraded - lastTransitionTime: "2019-08-21T14:27:15Z" message: "" reason: "" status: "False" type: Degraded - lastTransitionTime: "2019-08-21T14:34:51Z" message: All nodes are updated with rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c reason: "" status: "True" type: Updated - lastTransitionTime: "2019-08-21T14:34:51Z" message: "" reason: "" status: "False" type: Updating configuration: name: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c source: - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 00-worker - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 01-worker-container-runtime - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 01-worker-kubelet - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 99-worker-ssh degradedMachineCount: 0 machineCount: 3 observedGeneration: 2 readyMachineCount: 3 unavailableMachineCount: 0 updatedMachineCount: 3 $ oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-140-224.us-west-2.compute.internal Ready master 19m v1.14.0+f667219f4 ip-10-0-142-195.us-west-2.compute.internal Ready worker 12m v1.14.0+f667219f4 ip-10-0-152-89.us-west-2.compute.internal Ready master 19m v1.14.0+f667219f4 ip-10-0-157-197.us-west-2.compute.internal Ready worker 11m v1.14.0+f667219f4 ip-10-0-167-57.us-west-2.compute.internal Ready master 19m v1.14.0+f667219f4 ip-10-0-173-22.us-west-2.compute.internal Ready worker 12m v1.14.0+f667219f4 $ oc describe node ip-10-0-173-22.us-west-2.compute.internal Name: ip-10-0-173-22.us-west-2.compute.internal Roles: worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/instance-type=m5.large beta.kubernetes.io/os=linux failure-domain.beta.kubernetes.io/region=us-west-2 failure-domain.beta.kubernetes.io/zone=us-west-2c kubernetes.io/arch=amd64 kubernetes.io/hostname=ip-10-0-173-22 kubernetes.io/os=linux node-role.kubernetes.io/worker= node.openshift.io/os_id=rhcos Annotations: machine.openshift.io/machine: openshift-machine-api/mnguyen-w8rzg-worker-us-west-2c-8sfwr machineconfiguration.openshift.io/currentConfig: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c machineconfiguration.openshift.io/desiredConfig: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c machineconfiguration.openshift.io/state: Done volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Wed, 21 Aug 2019 10:32:46 -0400 Taints: <none> Unschedulable: false Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Wed, 21 Aug 2019 10:45:06 -0400 Wed, 21 Aug 2019 10:32:46 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Wed, 21 Aug 2019 10:45:06 -0400 Wed, 21 Aug 2019 10:32:46 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Wed, 21 Aug 2019 10:45:06 -0400 Wed, 21 Aug 2019 10:32:46 -0400 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Wed, 21 Aug 2019 10:45:06 -0400 Wed, 21 Aug 2019 10:33:46 -0400 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.0.173.22 InternalDNS: ip-10-0-173-22.us-west-2.compute.internal Hostname: ip-10-0-173-22.us-west-2.compute.internal Capacity: attachable-volumes-aws-ebs: 25 cpu: 2 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 7861176Ki pods: 250 Allocatable: attachable-volumes-aws-ebs: 25 cpu: 1500m hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 7246776Ki pods: 250 System Info: Machine ID: ec2c8fea0f6a0f036773cf2dcb8b4134 System UUID: ec2c8fea-0f6a-0f03-6773-cf2dcb8b4134 Boot ID: 5f627768-c1ff-473c-9a09-c2e6d5e9d8e9 Kernel Version: 4.18.0-80.7.2.el8_0.x86_64 OS Image: Red Hat Enterprise Linux CoreOS 42.80.20190816.2 (Ootpa) Operating System: linux Architecture: amd64 Container Runtime Version: cri-o://1.14.10-0.7.dev.rhaos4.2.git7e863dd.el8-dev Kubelet Version: v1.14.0+f667219f4 Kube-Proxy Version: v1.14.0+f667219f4 ProviderID: aws:///us-west-2c/i-0a105357bc54d2199 Non-terminated Pods: (14 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- openshift-cluster-node-tuning-operator tuned-wsgfp 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 12m openshift-dns dns-default-dl7hq 110m (7%) 0 (0%) 70Mi (0%) 512Mi (7%) 12m openshift-image-registry node-ca-dkhhs 10m (0%) 0 (0%) 10Mi (0%) 0 (0%) 11m openshift-machine-config-operator machine-config-daemon-wlns8 20m (1%) 0 (0%) 50Mi (0%) 0 (0%) 11m openshift-monitoring alertmanager-main-0 100m (6%) 100m (6%) 225Mi (3%) 25Mi (0%) 10m openshift-monitoring kube-state-metrics-6bdfd467f9-kmj2v 30m (2%) 0 (0%) 120Mi (1%) 0 (0%) 15m openshift-monitoring node-exporter-rdns2 10m (0%) 0 (0%) 20Mi (0%) 0 (0%) 12m openshift-monitoring openshift-state-metrics-675c468964-fbc9m 120m (8%) 0 (0%) 190Mi (2%) 0 (0%) 15m openshift-monitoring prometheus-adapter-65bf8db64c-t4lrm 10m (0%) 0 (0%) 20Mi (0%) 0 (0%) 10m openshift-monitoring prometheus-k8s-0 430m (28%) 200m (13%) 1134Mi (16%) 50Mi (0%) 10m openshift-monitoring prometheus-operator-8474479b67-kg98m 10m (0%) 0 (0%) 60Mi (0%) 0 (0%) 10m openshift-multus multus-t54rs 10m (0%) 0 (0%) 150Mi (2%) 0 (0%) 12m openshift-sdn ovs-th5fc 200m (13%) 0 (0%) 400Mi (5%) 0 (0%) 12m openshift-sdn sdn-tttfp 100m (6%) 0 (0%) 200Mi (2%) 0 (0%) 12m Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 1170m (78%) 300m (20%) memory 2699Mi (38%) 587Mi (8%) ephemeral-storage 0 (0%) 0 (0%) attachable-volumes-aws-ebs 0 0 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NodeHasSufficientMemory 12m (x8 over 13m) kubelet, ip-10-0-173-22.us-west-2.compute.internal Node ip-10-0-173-22.us-west-2.compute.internal status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 12m (x8 over 13m) kubelet, ip-10-0-173-22.us-west-2.compute.internal Node ip-10-0-173-22.us-west-2.compute.internal status is now: NodeHasNoDiskPressure $ oc edit node ip-10-0-173-22.us-west-2.compute.internal node/ip-10-0-173-22.us-west-2.compute.internal edited $ oc edit node ip-10-0-173-22.us-west-2.compute.internal ( s/node-role.kubernetes.io\/worker:/node-role.kubernetes.io\/infra:/ ) $ oc get node NAME STATUS ROLES AGE VERSION ip-10-0-140-224.us-west-2.compute.internal Ready master 20m v1.14.0+f667219f4 ip-10-0-142-195.us-west-2.compute.internal Ready worker 13m v1.14.0+f667219f4 ip-10-0-152-89.us-west-2.compute.internal Ready master 20m v1.14.0+f667219f4 ip-10-0-157-197.us-west-2.compute.internal Ready worker 13m v1.14.0+f667219f4 ip-10-0-167-57.us-west-2.compute.internal Ready master 20m v1.14.0+f667219f4 ip-10-0-173-22.us-west-2.compute.internal Ready infra 13m v1.14.0+f667219f4 $ oc describe machineconfigpool worker Name: worker Namespace: Labels: machineconfiguration.openshift.io/mco-built-in= Annotations: <none> API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfigPool Metadata: Creation Timestamp: 2019-08-21T14:26:46Z Generation: 2 Resource Version: 13234 Self Link: /apis/machineconfiguration.openshift.io/v1/machineconfigpools/worker UID: b46bbcb7-c41f-11e9-bb1a-0239f61aa074 Spec: Configuration: Name: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c Source: API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 00-worker API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 01-worker-container-runtime API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 01-worker-kubelet API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 99-worker-ssh Machine Config Selector: Match Labels: machineconfiguration.openshift.io/role: worker Node Selector: Match Labels: node-role.kubernetes.io/worker: Paused: false Status: Conditions: Last Transition Time: 2019-08-21T14:27:12Z Message: Reason: Status: False Type: RenderDegraded Last Transition Time: 2019-08-21T14:27:15Z Message: Reason: Status: False Type: NodeDegraded Last Transition Time: 2019-08-21T14:27:15Z Message: Reason: Status: False Type: Degraded Last Transition Time: 2019-08-21T14:34:51Z Message: All nodes are updated with rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c Reason: Status: True Type: Updated Last Transition Time: 2019-08-21T14:34:51Z Message: Reason: Status: False Type: Updating Configuration: Name: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c Source: API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 00-worker API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 01-worker-container-runtime API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 01-worker-kubelet API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 99-worker-ssh Degraded Machine Count: 0 Machine Count: 3 Observed Generation: 2 Ready Machine Count: 3 Unavailable Machine Count: 0 Updated Machine Count: 3 Events: <none> $ oc describe machineconfigpool infra Name: infra Namespace: Labels: <none> Annotations: <none> API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfigPool Metadata: Creation Timestamp: 2019-08-21T14:45:04Z Generation: 2 Resource Version: 17287 Self Link: /apis/machineconfiguration.openshift.io/v1/machineconfigpools/infra UID: 42ca997d-c422-11e9-9cf0-0ab5045af90e Spec: Configuration: Name: rendered-infra-0fbb7c0ce09edd8e04bf68881c95d62c Source: API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 00-worker API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 01-worker-container-runtime API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 01-worker-kubelet API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 99-worker-b46bbcb7-c41f-11e9-bb1a-0239f61aa074-registries API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 99-worker-ssh Machine Config Selector: Match Expressions: Key: machineconfiguration.openshift.io/role Operator: In Values: worker infra Node Selector: Match Labels: node-role.kubernetes.io/infra: Paused: false Status: Conditions: Last Transition Time: 2019-08-21T14:45:09Z Message: Reason: Status: False Type: RenderDegraded Last Transition Time: 2019-08-21T14:45:14Z Message: Reason: Status: False Type: NodeDegraded Last Transition Time: 2019-08-21T14:45:14Z Message: Reason: Status: False Type: Degraded Last Transition Time: 2019-08-21T14:47:58Z Message: Reason: Status: False Type: Updated Last Transition Time: 2019-08-21T14:47:58Z Message: All nodes are updating to rendered-infra-0fbb7c0ce09edd8e04bf68881c95d62c Reason: Status: True Type: Updating Configuration: Name: rendered-infra-0fbb7c0ce09edd8e04bf68881c95d62c Source: API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 00-worker API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 01-worker-container-runtime API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 01-worker-kubelet API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 99-worker-b46bbcb7-c41f-11e9-bb1a-0239f61aa074-registries API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 99-worker-ssh Degraded Machine Count: 0 Machine Count: 1 Observed Generation: 2 Ready Machine Count: 0 Unavailable Machine Count: 0 Updated Machine Count: 0 Events: <none> $ oc describe machineconfigpool worker Name: worker Namespace: Labels: machineconfiguration.openshift.io/mco-built-in= Annotations: <none> API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfigPool Metadata: Creation Timestamp: 2019-08-21T14:26:46Z Generation: 2 Resource Version: 17284 Self Link: /apis/machineconfiguration.openshift.io/v1/machineconfigpools/worker UID: b46bbcb7-c41f-11e9-bb1a-0239f61aa074 Spec: Configuration: Name: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c Source: API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 00-worker API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 01-worker-container-runtime API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 01-worker-kubelet API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 99-worker-ssh Machine Config Selector: Match Labels: machineconfiguration.openshift.io/role: worker Node Selector: Match Labels: node-role.kubernetes.io/worker: Paused: false Status: Conditions: Last Transition Time: 2019-08-21T14:27:12Z Message: Reason: Status: False Type: RenderDegraded Last Transition Time: 2019-08-21T14:27:15Z Message: Reason: Status: False Type: NodeDegraded Last Transition Time: 2019-08-21T14:27:15Z Message: Reason: Status: False Type: Degraded Last Transition Time: 2019-08-21T14:34:51Z Message: All nodes are updated with rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c Reason: Status: True Type: Updated Last Transition Time: 2019-08-21T14:34:51Z Message: Reason: Status: False Type: Updating Configuration: Name: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c Source: API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 00-worker API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 01-worker-container-runtime API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 01-worker-kubelet API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Name: 99-worker-ssh Degraded Machine Count: 0 Machine Count: 2 Observed Generation: 2 Ready Machine Count: 2 Unavailable Machine Count: 0 Updated Machine Count: 2 Events: <none>
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922
Hi I found this bug still exist in ocp4.5.4, before I have 6 worker nodes, but after I changed one of it to infra node, the machinecount of mcp is still 6 [student@workstation ~]$ oc get nodes -l node-role.kubernetes.io/worker NAME STATUS ROLES AGE VERSION worker01 Ready worker 151d v1.18.3+012b3ec worker02 Ready worker 151d v1.18.3+012b3ec worker03 Ready worker 151d v1.18.3+012b3ec worker04 Ready worker 19m v1.18.3+012b3ec worker05 Ready worker 19m v1.18.3+012b3ec [student@workstation ~]$ oc get mcp worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-ecc5564cb238470a57fcdb8c49a9bc29 True False False 6 6 6 0 151d The only way I found to reflect the new machinecount is to recreate the MCP [student@workstation ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.4 True False 151d Cluster version is 4.5.4