Bug 1738831
| Summary: | parent MCP doesn't update machine count when there's another custom pool for a subset of the parent MCP | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Antonio Murdaca <amurdaca> |
| Component: | Machine Config Operator | Assignee: | Antonio Murdaca <amurdaca> |
| Status: | CLOSED ERRATA | QA Contact: | Micah Abbott <miabbott> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.2.0 | CC: | mnguyen, nsu |
| Target Milestone: | --- | ||
| Target Release: | 4.2.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-10-16 06:35:15 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Antonio Murdaca
2019-08-08 09:02:30 UTC
Verified on 4.2.0-0.nightly-2019-08-20-162755
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.2.0-0.nightly-2019-08-20-162755 True False 11m Cluster version is 4.2.0-0.nightly-2019-08-20-162755
cat mcpinfra.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
name: infra
spec:
machineConfigSelector:
matchExpressions:
- {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]}
maxUnavailable: null
nodeSelector:
matchLabels:
node-role.kubernetes.io/infra: ""
paused: false
$ oc create -f mcpinfra.yaml
machineconfigpool.machineconfiguration.openshift.io/infra created
$ oc get machineconfigpool
NAME CONFIG UPDATED UPDATING DEGRADED
infra rendered-infra-0fbb7c0ce09edd8e04bf68881c95d62c True False False
master rendered-master-72b9921bcfbd8242bfb984850defee68 True False False
worker rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c True False False
$ oc get machineconfigpool worker -o yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
creationTimestamp: "2019-08-21T14:26:46Z"
generation: 2
labels:
machineconfiguration.openshift.io/mco-built-in: ""
name: worker
resourceVersion: "13234"
selfLink: /apis/machineconfiguration.openshift.io/v1/machineconfigpools/worker
uid: b46bbcb7-c41f-11e9-bb1a-0239f61aa074
spec:
configuration:
name: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c
source:
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 00-worker
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 01-worker-container-runtime
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 01-worker-kubelet
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 99-worker-ssh
machineConfigSelector:
matchLabels:
machineconfiguration.openshift.io/role: worker
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker: ""
paused: false
status:
conditions:
- lastTransitionTime: "2019-08-21T14:27:12Z"
message: ""
reason: ""
status: "False"
type: RenderDegraded
- lastTransitionTime: "2019-08-21T14:27:15Z"
message: ""
reason: ""
status: "False"
type: NodeDegraded
- lastTransitionTime: "2019-08-21T14:27:15Z"
message: ""
reason: ""
status: "False"
type: Degraded
- lastTransitionTime: "2019-08-21T14:34:51Z"
message: All nodes are updated with rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c
reason: ""
status: "True"
type: Updated
- lastTransitionTime: "2019-08-21T14:34:51Z"
message: ""
reason: ""
status: "False"
type: Updating
configuration:
name: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c
source:
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 00-worker
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 01-worker-container-runtime
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 01-worker-kubelet
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 99-worker-ssh
degradedMachineCount: 0
machineCount: 3
observedGeneration: 2
readyMachineCount: 3
unavailableMachineCount: 0
updatedMachineCount: 3
$ oc get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-140-224.us-west-2.compute.internal Ready master 19m v1.14.0+f667219f4
ip-10-0-142-195.us-west-2.compute.internal Ready worker 12m v1.14.0+f667219f4
ip-10-0-152-89.us-west-2.compute.internal Ready master 19m v1.14.0+f667219f4
ip-10-0-157-197.us-west-2.compute.internal Ready worker 11m v1.14.0+f667219f4
ip-10-0-167-57.us-west-2.compute.internal Ready master 19m v1.14.0+f667219f4
ip-10-0-173-22.us-west-2.compute.internal Ready worker 12m v1.14.0+f667219f4
$ oc describe node ip-10-0-173-22.us-west-2.compute.internal
Name: ip-10-0-173-22.us-west-2.compute.internal
Roles: worker
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=m5.large
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=us-west-2
failure-domain.beta.kubernetes.io/zone=us-west-2c
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-10-0-173-22
kubernetes.io/os=linux
node-role.kubernetes.io/worker=
node.openshift.io/os_id=rhcos
Annotations: machine.openshift.io/machine: openshift-machine-api/mnguyen-w8rzg-worker-us-west-2c-8sfwr
machineconfiguration.openshift.io/currentConfig: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c
machineconfiguration.openshift.io/desiredConfig: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c
machineconfiguration.openshift.io/state: Done
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 21 Aug 2019 10:32:46 -0400
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Wed, 21 Aug 2019 10:45:06 -0400 Wed, 21 Aug 2019 10:32:46 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 21 Aug 2019 10:45:06 -0400 Wed, 21 Aug 2019 10:32:46 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 21 Aug 2019 10:45:06 -0400 Wed, 21 Aug 2019 10:32:46 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 21 Aug 2019 10:45:06 -0400 Wed, 21 Aug 2019 10:33:46 -0400 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.0.173.22
InternalDNS: ip-10-0-173-22.us-west-2.compute.internal
Hostname: ip-10-0-173-22.us-west-2.compute.internal
Capacity:
attachable-volumes-aws-ebs: 25
cpu: 2
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7861176Ki
pods: 250
Allocatable:
attachable-volumes-aws-ebs: 25
cpu: 1500m
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7246776Ki
pods: 250
System Info:
Machine ID: ec2c8fea0f6a0f036773cf2dcb8b4134
System UUID: ec2c8fea-0f6a-0f03-6773-cf2dcb8b4134
Boot ID: 5f627768-c1ff-473c-9a09-c2e6d5e9d8e9
Kernel Version: 4.18.0-80.7.2.el8_0.x86_64
OS Image: Red Hat Enterprise Linux CoreOS 42.80.20190816.2 (Ootpa)
Operating System: linux
Architecture: amd64
Container Runtime Version: cri-o://1.14.10-0.7.dev.rhaos4.2.git7e863dd.el8-dev
Kubelet Version: v1.14.0+f667219f4
Kube-Proxy Version: v1.14.0+f667219f4
ProviderID: aws:///us-west-2c/i-0a105357bc54d2199
Non-terminated Pods: (14 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
openshift-cluster-node-tuning-operator tuned-wsgfp 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 12m
openshift-dns dns-default-dl7hq 110m (7%) 0 (0%) 70Mi (0%) 512Mi (7%) 12m
openshift-image-registry node-ca-dkhhs 10m (0%) 0 (0%) 10Mi (0%) 0 (0%) 11m
openshift-machine-config-operator machine-config-daemon-wlns8 20m (1%) 0 (0%) 50Mi (0%) 0 (0%) 11m
openshift-monitoring alertmanager-main-0 100m (6%) 100m (6%) 225Mi (3%) 25Mi (0%) 10m
openshift-monitoring kube-state-metrics-6bdfd467f9-kmj2v 30m (2%) 0 (0%) 120Mi (1%) 0 (0%) 15m
openshift-monitoring node-exporter-rdns2 10m (0%) 0 (0%) 20Mi (0%) 0 (0%) 12m
openshift-monitoring openshift-state-metrics-675c468964-fbc9m 120m (8%) 0 (0%) 190Mi (2%) 0 (0%) 15m
openshift-monitoring prometheus-adapter-65bf8db64c-t4lrm 10m (0%) 0 (0%) 20Mi (0%) 0 (0%) 10m
openshift-monitoring prometheus-k8s-0 430m (28%) 200m (13%) 1134Mi (16%) 50Mi (0%) 10m
openshift-monitoring prometheus-operator-8474479b67-kg98m 10m (0%) 0 (0%) 60Mi (0%) 0 (0%) 10m
openshift-multus multus-t54rs 10m (0%) 0 (0%) 150Mi (2%) 0 (0%) 12m
openshift-sdn ovs-th5fc 200m (13%) 0 (0%) 400Mi (5%) 0 (0%) 12m
openshift-sdn sdn-tttfp 100m (6%) 0 (0%) 200Mi (2%) 0 (0%) 12m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 1170m (78%) 300m (20%)
memory 2699Mi (38%) 587Mi (8%)
ephemeral-storage 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasSufficientMemory 12m (x8 over 13m) kubelet, ip-10-0-173-22.us-west-2.compute.internal Node ip-10-0-173-22.us-west-2.compute.internal status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 12m (x8 over 13m) kubelet, ip-10-0-173-22.us-west-2.compute.internal Node ip-10-0-173-22.us-west-2.compute.internal status is now: NodeHasNoDiskPressure
$ oc edit node ip-10-0-173-22.us-west-2.compute.internal
node/ip-10-0-173-22.us-west-2.compute.internal edited
$ oc edit node ip-10-0-173-22.us-west-2.compute.internal
( s/node-role.kubernetes.io\/worker:/node-role.kubernetes.io\/infra:/ )
$ oc get node
NAME STATUS ROLES AGE VERSION
ip-10-0-140-224.us-west-2.compute.internal Ready master 20m v1.14.0+f667219f4
ip-10-0-142-195.us-west-2.compute.internal Ready worker 13m v1.14.0+f667219f4
ip-10-0-152-89.us-west-2.compute.internal Ready master 20m v1.14.0+f667219f4
ip-10-0-157-197.us-west-2.compute.internal Ready worker 13m v1.14.0+f667219f4
ip-10-0-167-57.us-west-2.compute.internal Ready master 20m v1.14.0+f667219f4
ip-10-0-173-22.us-west-2.compute.internal Ready infra 13m v1.14.0+f667219f4
$ oc describe machineconfigpool worker
Name: worker
Namespace:
Labels: machineconfiguration.openshift.io/mco-built-in=
Annotations: <none>
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfigPool
Metadata:
Creation Timestamp: 2019-08-21T14:26:46Z
Generation: 2
Resource Version: 13234
Self Link: /apis/machineconfiguration.openshift.io/v1/machineconfigpools/worker
UID: b46bbcb7-c41f-11e9-bb1a-0239f61aa074
Spec:
Configuration:
Name: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c
Source:
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 00-worker
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-container-runtime
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-kubelet
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-worker-ssh
Machine Config Selector:
Match Labels:
machineconfiguration.openshift.io/role: worker
Node Selector:
Match Labels:
node-role.kubernetes.io/worker:
Paused: false
Status:
Conditions:
Last Transition Time: 2019-08-21T14:27:12Z
Message:
Reason:
Status: False
Type: RenderDegraded
Last Transition Time: 2019-08-21T14:27:15Z
Message:
Reason:
Status: False
Type: NodeDegraded
Last Transition Time: 2019-08-21T14:27:15Z
Message:
Reason:
Status: False
Type: Degraded
Last Transition Time: 2019-08-21T14:34:51Z
Message: All nodes are updated with rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c
Reason:
Status: True
Type: Updated
Last Transition Time: 2019-08-21T14:34:51Z
Message:
Reason:
Status: False
Type: Updating
Configuration:
Name: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c
Source:
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 00-worker
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-container-runtime
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-kubelet
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-worker-ssh
Degraded Machine Count: 0
Machine Count: 3
Observed Generation: 2
Ready Machine Count: 3
Unavailable Machine Count: 0
Updated Machine Count: 3
Events: <none>
$ oc describe machineconfigpool infra
Name: infra
Namespace:
Labels: <none>
Annotations: <none>
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfigPool
Metadata:
Creation Timestamp: 2019-08-21T14:45:04Z
Generation: 2
Resource Version: 17287
Self Link: /apis/machineconfiguration.openshift.io/v1/machineconfigpools/infra
UID: 42ca997d-c422-11e9-9cf0-0ab5045af90e
Spec:
Configuration:
Name: rendered-infra-0fbb7c0ce09edd8e04bf68881c95d62c
Source:
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 00-worker
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-container-runtime
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-kubelet
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-worker-b46bbcb7-c41f-11e9-bb1a-0239f61aa074-registries
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-worker-ssh
Machine Config Selector:
Match Expressions:
Key: machineconfiguration.openshift.io/role
Operator: In
Values:
worker
infra
Node Selector:
Match Labels:
node-role.kubernetes.io/infra:
Paused: false
Status:
Conditions:
Last Transition Time: 2019-08-21T14:45:09Z
Message:
Reason:
Status: False
Type: RenderDegraded
Last Transition Time: 2019-08-21T14:45:14Z
Message:
Reason:
Status: False
Type: NodeDegraded
Last Transition Time: 2019-08-21T14:45:14Z
Message:
Reason:
Status: False
Type: Degraded
Last Transition Time: 2019-08-21T14:47:58Z
Message:
Reason:
Status: False
Type: Updated
Last Transition Time: 2019-08-21T14:47:58Z
Message: All nodes are updating to rendered-infra-0fbb7c0ce09edd8e04bf68881c95d62c
Reason:
Status: True
Type: Updating
Configuration:
Name: rendered-infra-0fbb7c0ce09edd8e04bf68881c95d62c
Source:
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 00-worker
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-container-runtime
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-kubelet
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-worker-b46bbcb7-c41f-11e9-bb1a-0239f61aa074-registries
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-worker-ssh
Degraded Machine Count: 0
Machine Count: 1
Observed Generation: 2
Ready Machine Count: 0
Unavailable Machine Count: 0
Updated Machine Count: 0
Events: <none>
$ oc describe machineconfigpool worker
Name: worker
Namespace:
Labels: machineconfiguration.openshift.io/mco-built-in=
Annotations: <none>
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfigPool
Metadata:
Creation Timestamp: 2019-08-21T14:26:46Z
Generation: 2
Resource Version: 17284
Self Link: /apis/machineconfiguration.openshift.io/v1/machineconfigpools/worker
UID: b46bbcb7-c41f-11e9-bb1a-0239f61aa074
Spec:
Configuration:
Name: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c
Source:
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 00-worker
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-container-runtime
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-kubelet
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-worker-ssh
Machine Config Selector:
Match Labels:
machineconfiguration.openshift.io/role: worker
Node Selector:
Match Labels:
node-role.kubernetes.io/worker:
Paused: false
Status:
Conditions:
Last Transition Time: 2019-08-21T14:27:12Z
Message:
Reason:
Status: False
Type: RenderDegraded
Last Transition Time: 2019-08-21T14:27:15Z
Message:
Reason:
Status: False
Type: NodeDegraded
Last Transition Time: 2019-08-21T14:27:15Z
Message:
Reason:
Status: False
Type: Degraded
Last Transition Time: 2019-08-21T14:34:51Z
Message: All nodes are updated with rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c
Reason:
Status: True
Type: Updated
Last Transition Time: 2019-08-21T14:34:51Z
Message:
Reason:
Status: False
Type: Updating
Configuration:
Name: rendered-worker-0fbb7c0ce09edd8e04bf68881c95d62c
Source:
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 00-worker
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-container-runtime
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-kubelet
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-worker-ssh
Degraded Machine Count: 0
Machine Count: 2
Observed Generation: 2
Ready Machine Count: 2
Unavailable Machine Count: 0
Updated Machine Count: 2
Events: <none>
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922 Hi I found this bug still exist in ocp4.5.4, before I have 6 worker nodes, but after I changed one of it to infra node, the machinecount of mcp is still 6 [student@workstation ~]$ oc get nodes -l node-role.kubernetes.io/worker NAME STATUS ROLES AGE VERSION worker01 Ready worker 151d v1.18.3+012b3ec worker02 Ready worker 151d v1.18.3+012b3ec worker03 Ready worker 151d v1.18.3+012b3ec worker04 Ready worker 19m v1.18.3+012b3ec worker05 Ready worker 19m v1.18.3+012b3ec [student@workstation ~]$ oc get mcp worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-ecc5564cb238470a57fcdb8c49a9bc29 True False False 6 6 6 0 151d The only way I found to reflect the new machinecount is to recreate the MCP [student@workstation ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.4 True False 151d Cluster version is 4.5.4 |