Bug 1880136
| Summary: | Cluster Autoscaler on OCP 4.6 RHV-IPI keeps on deleting existing and adding new nodes | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Apoorva Jagtap <apjagtap> |
| Component: | Cloud Compute | Assignee: | Gal Zaidman <gzaidman> |
| Cloud Compute sub component: | oVirt Provider | QA Contact: | Lucie Leistnerova <lleistne> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | aos-bugs, apjagtap, eparis, gzaidman, jokerman, jzmeskal, mimccune, vpagar |
| Version: | 4.6 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.7.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-10-01 07:06:24 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Possiable dup of https://bugzilla.redhat.com/show_bug.cgi?id=1881051 Targeting to 4.7 since we will probably not be able to support autoscalling on 4.6. I can confirm that I replicated this with 4.6.0-0.nightly-2020-09-10-145837.
Steps:
1. Have OCP with 3 masters and 3 workers
2. cat cluster_autoscaler.yaml
apiVersion: "autoscaling.openshift.io/v1"
kind: "ClusterAutoscaler"
metadata:
name: "default"
spec:
podPriorityThreshold: -10
resourceLimits:
maxNodesTotal: 9
cores:
min: 24
max: 40
memory:
min: 96
max: 256
scaleDown:
enabled: true
delayAfterAdd: 10s
delayAfterDelete: 10s
delayAfterFailure: 30s
unneededTime: 30s
3. oc create -f cluster_autoscaler.yaml
4. cat machine_autoscaler.yaml
apiVersion: "autoscaling.openshift.io/v1beta1"
kind: "MachineAutoscaler"
metadata:
name: "primary-jnzvt-worker-0"
namespace: "openshift-machine-api"
spec:
minReplicas: 3
maxReplicas: 6
scaleTargetRef:
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
name: primary-jnzvt-worker-0
5. oc create -f machine_autoscaler.yaml
6. oc apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: autoscaler-demo
EOF
7. cat scale-up.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: scale-up
labels:
app: scale-up
spec:
replicas: 20
selector:
matchLabels:
app: scale-up
template:
metadata:
labels:
app: scale-up
spec:
containers:
- name: origin-base
image: openshift/origin-base
resources:
requests:
memory: 2Gi
command:
- /bin/sh
- "-c"
- "echo 'this should be in the logs' && sleep 86400"
terminationGracePeriodSeconds: 0
8. oc apply -n autoscaler-demo -f scale-up.yaml
Results: At first, the OCP cluster attempted to schedule all the 20 pods and correctly realized it does not have enough resources. Therefore it scaled up the worker MachineSet by adding one new worker. For a while, everything was fine. Instead of original 3 workers, I had a cluster with 4 workers and all the 20 requested pods were running. However, after some minutes, the cluster started deleting existing VMs and provisioning new ones. See these outputs taken couple of minutes apart from each other.
1. This is the beginning. Cluster quickly realizes it needs an additional node to be able to schedule all the pods.
oc get pods -n autoscaler-demo && oc get machine -n openshift-machine-api
NAME READY STATUS RESTARTS AGE
scale-up-5fd5c67f64-7p4rm 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-9qsr7 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-bjc4b 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-c9cx6 0/1 Pending 0 13s
scale-up-5fd5c67f64-dr4v2 0/1 Pending 0 13s
scale-up-5fd5c67f64-fnqvc 0/1 Pending 0 13s
scale-up-5fd5c67f64-ftlqd 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-gwtrt 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-kzw9f 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-l4j88 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-mq9kf 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-pktpx 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-qtcdk 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-rvkpc 0/1 Pending 0 13s
scale-up-5fd5c67f64-spfnb 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-tfkr7 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-ws4kf 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-xjmht 0/1 Pending 0 13s
scale-up-5fd5c67f64-xlz7n 0/1 ContainerCreating 0 13s
scale-up-5fd5c67f64-znd8b 0/1 ContainerCreating 0 13s
NAME PHASE TYPE REGION ZONE AGE
primary-jnzvt-master-0 Running 14h
primary-jnzvt-master-1 Running 14h
primary-jnzvt-master-2 Running 14h
primary-jnzvt-worker-0-nfdg4 Running 14h
primary-jnzvt-worker-0-q2vjs Running 14h
primary-jnzvt-worker-0-r2m57 Running 14h
primary-jnzvt-worker-0-vxcjh Provisioning 4s
2. After some time, new worker machine is up and all the pods are running.
oc get pods -n autoscaler-demo && oc get machine -n openshift-machine-api
NAME READY STATUS RESTARTS AGE
scale-up-5fd5c67f64-7p4rm 1/1 Running 0 17m
scale-up-5fd5c67f64-9qsr7 1/1 Running 0 17m
scale-up-5fd5c67f64-bjc4b 1/1 Running 0 17m
scale-up-5fd5c67f64-c9cx6 1/1 Running 0 17m
scale-up-5fd5c67f64-dr4v2 1/1 Running 0 17m
scale-up-5fd5c67f64-fnqvc 1/1 Running 0 17m
scale-up-5fd5c67f64-ftlqd 1/1 Running 0 17m
scale-up-5fd5c67f64-gwtrt 1/1 Running 0 17m
scale-up-5fd5c67f64-kzw9f 1/1 Running 0 17m
scale-up-5fd5c67f64-l4j88 1/1 Running 0 17m
scale-up-5fd5c67f64-mq9kf 1/1 Running 0 17m
scale-up-5fd5c67f64-pktpx 1/1 Running 0 17m
scale-up-5fd5c67f64-qtcdk 1/1 Running 0 17m
scale-up-5fd5c67f64-rvkpc 1/1 Running 0 17m
scale-up-5fd5c67f64-spfnb 1/1 Running 0 17m
scale-up-5fd5c67f64-tfkr7 1/1 Running 0 17m
scale-up-5fd5c67f64-ws4kf 1/1 Running 0 17m
scale-up-5fd5c67f64-xjmht 1/1 Running 0 17m
scale-up-5fd5c67f64-xlz7n 1/1 Running 0 17m
scale-up-5fd5c67f64-znd8b 1/1 Running 0 17m
NAME PHASE TYPE REGION ZONE AGE
primary-jnzvt-master-0 Running 14h
primary-jnzvt-master-1 Running 14h
primary-jnzvt-master-2 Running 14h
primary-jnzvt-worker-0-nfdg4 Running 14h
primary-jnzvt-worker-0-q2vjs Running 14h
primary-jnzvt-worker-0-r2m57 Running 14h
primary-jnzvt-worker-0-vxcjh Running 17m
3. After some minutes, I realized some pods are again in Pending status. For some reason, OCP started deleteing existing machines and provisioning new ones.
oc get pods -n autoscaler-demo
NAME READY STATUS RESTARTS AGE
scale-up-5fd5c67f64-75dls 0/1 Pending 0 23s
scale-up-5fd5c67f64-7p4rm 1/1 Running 0 25m
scale-up-5fd5c67f64-9qsr7 1/1 Running 0 25m
scale-up-5fd5c67f64-c9cx6 1/1 Running 0 25m
scale-up-5fd5c67f64-cjksm 0/1 Pending 0 24s
scale-up-5fd5c67f64-dr4v2 1/1 Running 0 25m
scale-up-5fd5c67f64-fnqvc 1/1 Running 0 25m
scale-up-5fd5c67f64-gwtrt 1/1 Running 0 25m
scale-up-5fd5c67f64-h6klv 0/1 Pending 0 50s
scale-up-5fd5c67f64-hrxvs 0/1 Pending 0 23s
scale-up-5fd5c67f64-j5hbb 0/1 Pending 0 50s
scale-up-5fd5c67f64-lf278 0/1 Pending 0 50s
scale-up-5fd5c67f64-lqlft 0/1 Pending 0 50s
scale-up-5fd5c67f64-qtcdk 1/1 Running 0 25m
scale-up-5fd5c67f64-rrj4g 0/1 Pending 0 23s
scale-up-5fd5c67f64-rvkpc 1/1 Running 0 25m
scale-up-5fd5c67f64-xjmht 1/1 Running 0 25m
scale-up-5fd5c67f64-xlz7n 1/1 Running 0 25m
scale-up-5fd5c67f64-zbh9c 0/1 Pending 0 23s
scale-up-5fd5c67f64-zfg82 0/1 Pending 0 50s
[root@ocp-qe-1 primary_jzmeskal]# oc get mahcine
error: the server doesn't have a resource type "mahcine"
[root@ocp-qe-1 primary_jzmeskal]# oc get machine -n openshift-machine-api
NAME PHASE TYPE REGION ZONE AGE
primary-jnzvt-master-0 Running 14h
primary-jnzvt-master-1 Running 14h
primary-jnzvt-master-2 Running 14h
primary-jnzvt-worker-0-7jhbz 27s
primary-jnzvt-worker-0-9bflw 0s
primary-jnzvt-worker-0-gxn8c 27s
primary-jnzvt-worker-0-nfdg4 Deleting 14h
primary-jnzvt-worker-0-pjp7z Provisioning 52s
primary-jnzvt-worker-0-q2vjs Running 14h
primary-jnzvt-worker-0-vxcjh Running 25m
primary-jnzvt-worker-0-wff7x 0s
4. This kept on going. Notice here that not only the newly created worker (vxcjh) is being deleted, but one of the original ones (q2vjs) as well.
oc get machine
NAME PHASE TYPE REGION ZONE AGE
primary-jnzvt-master-0 Running 14h
primary-jnzvt-master-1 Running 14h
primary-jnzvt-master-2 Running 14h
primary-jnzvt-worker-0-7jhbz Provisioning 7m12s
primary-jnzvt-worker-0-9bflw 6m45s
primary-jnzvt-worker-0-gxn8c Provisioning 7m12s
primary-jnzvt-worker-0-pjp7z Provisioned 7m37s
primary-jnzvt-worker-0-q2vjs Deleting 14h
primary-jnzvt-worker-0-vxcjh Deleting 31m
primary-jnzvt-worker-0-wff7x 6m45s
5. One of the original worker machines actually seems to be stuck in Deleting forever.
oc get machine
NAME PHASE TYPE REGION ZONE AGE
primary-jnzvt-master-0 Running 14h
primary-jnzvt-master-1 Running 14h
primary-jnzvt-master-2 Running 14h
primary-jnzvt-worker-0-7jhbz Provisioning 12m
primary-jnzvt-worker-0-9bflw Provisioning 11m
primary-jnzvt-worker-0-gxn8c Provisioning 12m
primary-jnzvt-worker-0-pjp7z Provisioned 12m
primary-jnzvt-worker-0-q2vjs Deleting 14h
primary-jnzvt-worker-0-wff7x Provisioning 11m
6. When I stopped waiting for the next developments, the cluster was in quite a bad state.
oc get pods -n autoscaler-demo && oc get machine -n openshift-machine-api
NAME READY STATUS RESTARTS AGE
scale-up-5fd5c67f64-544jl 0/1 Pending 0 13m
scale-up-5fd5c67f64-5mmgg 0/1 Pending 0 13m
scale-up-5fd5c67f64-75dls 0/1 Pending 0 15m
scale-up-5fd5c67f64-7fwnp 0/1 Pending 0 12m
scale-up-5fd5c67f64-7pj47 0/1 Pending 0 12m
scale-up-5fd5c67f64-8lt2p 0/1 Pending 0 12m
scale-up-5fd5c67f64-b9p4p 0/1 Pending 0 13m
scale-up-5fd5c67f64-cjksm 0/1 Pending 0 15m
scale-up-5fd5c67f64-h6klv 0/1 Pending 0 15m
scale-up-5fd5c67f64-hrxvs 0/1 Pending 0 15m
scale-up-5fd5c67f64-j5hbb 0/1 Pending 0 15m
scale-up-5fd5c67f64-jxjtp 0/1 Pending 0 13m
scale-up-5fd5c67f64-lf278 0/1 Pending 0 15m
scale-up-5fd5c67f64-lqlft 0/1 Pending 0 15m
scale-up-5fd5c67f64-pb2tf 0/1 Pending 0 13m
scale-up-5fd5c67f64-pdlhb 0/1 Pending 0 12m
scale-up-5fd5c67f64-rrj4g 0/1 Pending 0 15m
scale-up-5fd5c67f64-xctjz 0/1 Pending 0 13m
scale-up-5fd5c67f64-zbh9c 0/1 Pending 0 15m
scale-up-5fd5c67f64-zfg82 0/1 Pending 0 15m
NAME PHASE TYPE REGION ZONE AGE
primary-jnzvt-master-0 Running 15h
primary-jnzvt-master-1 Running 15h
primary-jnzvt-master-2 Running 15h
primary-jnzvt-worker-0-7jhbz Running 14m
primary-jnzvt-worker-0-9bflw Provisioning 14m
primary-jnzvt-worker-0-gxn8c Running 14m
primary-jnzvt-worker-0-pjp7z Provisioned 15m
primary-jnzvt-worker-0-q2vjs Deleting 14h
primary-jnzvt-worker-0-wff7x Provisioning 14m
Teardown: After all this, I tried to clean up resources.
1. oc delete -f scale-up.yaml -n autoscaler-demo
deployment.apps "scale-up" deleted
2. oc delete machineautoscaler primary-jnzvt-worker-0
machineautoscaler.autoscaling.openshift.io "primary-jnzvt-worker-0" deleted
3. oc delete clusterautoscaler default
clusterautoscaler.autoscaling.openshift.io "default" deleted
4. A lot of workers got stuck in bad states.
oc get machine
NAME PHASE TYPE REGION ZONE AGE
primary-jnzvt-master-0 Running 15h
primary-jnzvt-master-1 Running 15h
primary-jnzvt-master-2 Running 15h
primary-jnzvt-worker-0-7jhbz Running 34m
primary-jnzvt-worker-0-9bflw Provisioning 34m
primary-jnzvt-worker-0-gxn8c Running 34m
primary-jnzvt-worker-0-pjp7z Provisioned 35m
primary-jnzvt-worker-0-wff7x Provisioning 34m
5. oc scale --replicas=2 machineset primary-jnzvt-worker-0
machineset.machine.openshift.io/primary-jnzvt-worker-0 scaled
6. The scale-down of machines did not help, workers got stuck in bad states.
oc get machine
NAME PHASE TYPE REGION ZONE AGE
primary-jnzvt-master-0 Running 15h
primary-jnzvt-master-1 Running 15h
primary-jnzvt-master-2 Running 15h
primary-jnzvt-worker-0-gxn8c Deleting 55m
primary-jnzvt-worker-0-pjp7z Provisioned 55m
primary-jnzvt-worker-0-wff7x Provisioning 54m
(In reply to Gal Zaidman from comment #2) > Possiable dup of https://bugzilla.redhat.com/show_bug.cgi?id=1881051 ++, this seems like a duplicate also, i looked at the cluster-autoscaler logs and it looks like the same root cause. the cluster-autoscaler is unable to properly process the machines from the ovirt provider because the providerID is not being configured properly in the machine resources. ``` I0917 18:15:47.282252 1 static_autoscaler.go:320] 1 unregistered nodes present I0917 18:15:47.282432 1 static_autoscaler.go:592] Removing unregistered node 62b66c93-389c-4bcf-aa2d-7f6689e11f72 I0917 18:15:47.592062 1 static_autoscaler.go:328] Some unregistered nodes were removed, skipping iteration W0917 18:15:58.095319 1 clusterapi_controller.go:440] Machine "ocprhv-ptbf8-worker-1-l52vf" has no providerID W0917 18:15:58.095410 1 clusterapi_controller.go:440] Machine "ocprhv-ptbf8-worker-1-r4ddj" has no providerID W0917 18:15:58.095425 1 clusterapi_controller.go:440] Machine "ocprhv-ptbf8-worker-1-8xv4w" has no providerID ``` if the autoscaler does not have these providerIDs it will attempt to remove and replace those machines if it can (eg it's not at minimum size or hitting some cluster limit). there might be a synchronization issue with the providerID being properly populated for the cluster-autoscaler. i do see that in the original output the providerIDs appear for some of the machines. I reproduced this even without CPU and memory limits specification.
cat cluster_autoscaler.yml
apiVersion: "autoscaling.openshift.io/v1"
kind: "ClusterAutoscaler"
metadata:
name: "default"
spec:
podPriorityThreshold: -10
resourceLimits:
maxNodesTotal: 9
scaleDown:
enabled: true
delayAfterAdd: 10s
delayAfterDelete: 10s
delayAfterFailure: 30s
unneededTime: 30s
This bug is caused by what we are describing in 1883978 *** This bug has been marked as a duplicate of bug 1883978 *** |
Description of problem: Cluster Autoscaler on RHV-IPI OCP 4.6 does not work as expected. When load on the cluster increases, the autoscaler deletes the existing node from the cluster and adds up new nodes. Deletion of existing cluster nodes impacts other already scheduled pods of the cluster. Version-Release number of selected component (if applicable): $ oc version Client Version: 4.6.0-fc.4 Server Version: 4.6.0-fc.4 Kubernetes Version: v4.6.0-202009031349.p0-dirty How reproducible: Always Steps to Reproduce: 1. Install OCP 4.6 as RHV-IPI. 2. Configure the cluster-autoscaler and machine-autoscaler in the cluster using following YAML definitions: ##### Cluster autoscaler YAML definition: apiVersion: "autoscaling.openshift.io/v1" kind: "ClusterAutoscaler" metadata: name: "default" spec: scaleDown: enabled: true delayAfterAdd: 10s delayAfterDelete: 10s delayAfterFailure: 10s unneededTime: 10s ##### Machine-autoscaler YAML definition: apiVersion: "autoscaling.openshift.io/v1beta1" kind: "MachineAutoscaler" metadata: name: "worker-autoscale" namespace: "openshift-machine-api" spec: minReplicas: 2 maxReplicas: 4 scaleTargetRef: apiVersion: machine.openshift.io/v1beta1 kind: MachineSet name: ocprhv-ptbf8-worker-1 <<===== using ‘ocprhv-ptbf8-worker-1’ machineset as reference kind (out of two machinesets present) ###### Machinesets available in the cluster: # oc get machinesets NAME DESIRED CURRENT READY AVAILABLE AGE ocprhv-ptbf8-worker-0 1 1 1 1 7d9h ocprhv-ptbf8-worker-1 1 1 1 1 7d1h 3. Increase the load on the cluster using the Test scenario from documentation" https://docs.openshift.com/container-platform/3.11/admin_guide/cluster-autoscaler.html#testing-AWS-cluster-auto-scaler_cluster-auto-scaler Actual results: When load on the cluster increases, the autoscaler keeps on deleting the existing node belonging to the respective machineset from the cluster and adds up new nodes continuosly. Expected results: No existing node should be deleted, and only new nodes should be added as per the requirement, when high load is observed within the cluster. Additional info: [*] Existing nodes, and nodes being provisioned after high load observed on the cluster: ~~~ # oc get node NAME STATUS ROLES AGE VERSION ocprhv-ptbf8-master-0 Ready master 7d9h v1.19.0-rc.2+514f31a ocprhv-ptbf8-master-1 Ready master 7d9h v1.19.0-rc.2+514f31a ocprhv-ptbf8-master-2 Ready master 7d9h v1.19.0-rc.2+514f31a ocprhv-ptbf8-worker-0-qhrx5 Ready worker 7d8h v1.19.0-rc.2+514f31a ocprhv-ptbf8-worker-1-ldg58 Ready worker 24m v1.19.0-rc.2+514f31a # oc get machines -owide NAME PHASE TYPE REGION ZONE AGE NODE PROVIDERID STATE ocprhv-ptbf8-master-0 Running 7d10h ocprhv-ptbf8-master-0 44aea363-c806-4d5b-894f-dbb1bb3ae41d up ocprhv-ptbf8-master-1 Running 7d10h ocprhv-ptbf8-master-1 c6fe0dc3-99ac-4685-8e0a-8eb0a43246c3 up ocprhv-ptbf8-master-2 Running 7d10h ocprhv-ptbf8-master-2 13eedf50-722d-4147-a539-7dcb4a87c504 up ocprhv-ptbf8-worker-0-qhrx5 Running 7d9h ocprhv-ptbf8-worker-0-qhrx5 3a7f64b3-f2ad-4314-b6b4-2a6365fea608 up ocprhv-ptbf8-worker-1-8xv4w 16s ocprhv-ptbf8-worker-1-chbls Provisioning 41s ocprhv-ptbf8-worker-1-dqrth 41s ocprhv-ptbf8-worker-1-ldg58 Running 39m ocprhv-ptbf8-worker-1-ldg58 dcb1a82e-d592-4d4c-808e-9fa64de3aa49 up ocprhv-ptbf8-worker-1-xs6d9 16s ~~~ [*] Soon, it starts deleting the existing worker node 'ocprhv-ptbf8-worker-1-ldg58' (which belongs to machineset ocprhv-ptbf8-worker-1) ~~~ # oc get node NAME STATUS ROLES AGE VERSION ocprhv-ptbf8-master-0 Ready master 7d9h v1.19.0-rc.2+514f31a ocprhv-ptbf8-master-1 Ready master 7d9h v1.19.0-rc.2+514f31a ocprhv-ptbf8-master-2 Ready master 7d9h v1.19.0-rc.2+514f31a ocprhv-ptbf8-worker-0-qhrx5 Ready worker 7d8h v1.19.0-rc.2+514f31a ocprhv-ptbf8-worker-1-ldg58 Ready,SchedulingDisabled worker 27m v1.19.0-rc.2+514f31a # oc get machines -owide NAME PHASE TYPE REGION ZONE AGE NODE PROVIDERID STATE ocprhv-ptbf8-master-0 Running 7d10h ocprhv-ptbf8-master-0 44aea363-c806-4d5b-894f-dbb1bb3ae41d up ocprhv-ptbf8-master-1 Running 7d10h ocprhv-ptbf8-master-1 c6fe0dc3-99ac-4685-8e0a-8eb0a43246c3 up ocprhv-ptbf8-master-2 Running 7d10h ocprhv-ptbf8-master-2 13eedf50-722d-4147-a539-7dcb4a87c504 up ocprhv-ptbf8-worker-0-qhrx5 Running 7d9h ocprhv-ptbf8-worker-0-qhrx5 3a7f64b3-f2ad-4314-b6b4-2a6365fea608 up ocprhv-ptbf8-worker-1-8xv4w 3m53s ocprhv-ptbf8-worker-1-chbls Provisioning 4m18s ocprhv-ptbf8-worker-1-dqrth Provisioning 4m18s ocprhv-ptbf8-worker-1-ldg58 Deleting 43m ocprhv-ptbf8-worker-1-ldg58 dcb1a82e-d592-4d4c-808e-9fa64de3aa49 up ocprhv-ptbf8-worker-1-xs6d9 3m53s ~~~ [*] Further, the process of adding new and deleting the existing nodes continues: ~~~ # oc get nodes -w NAME STATUS ROLES AGE VERSION ocprhv-ptbf8-master-0 Ready master 7d10h v1.19.0-rc.2+514f31a ocprhv-ptbf8-master-1 Ready master 7d10h v1.19.0-rc.2+514f31a ocprhv-ptbf8-master-2 Ready master 7d10h v1.19.0-rc.2+514f31a ocprhv-ptbf8-worker-0-qhrx5 Ready worker 7d8h v1.19.0-rc.2+514f31a ocprhv-ptbf8-worker-1-chbls Ready,SchedulingDisabled worker 31m v1.19.0-rc.2+514f31a ocprhv-ptbf8-worker-1-xs6d9 Ready,SchedulingDisabled worker 20m v1.19.0-rc.2+514f31a # oc get machines -owide -n openshift-machine-api NAME PHASE TYPE REGION ZONE AGE NODE PROVIDERID STATE ocprhv-ptbf8-master-0 Running 7d11h ocprhv-ptbf8-master-0 44aea363-c806-4d5b-894f-dbb1bb3ae41d up ocprhv-ptbf8-master-1 Running 7d11h ocprhv-ptbf8-master-1 c6fe0dc3-99ac-4685-8e0a-8eb0a43246c3 up ocprhv-ptbf8-master-2 Running 7d11h ocprhv-ptbf8-master-2 13eedf50-722d-4147-a539-7dcb4a87c504 up ocprhv-ptbf8-worker-0-qhrx5 Running 7d9h ocprhv-ptbf8-worker-0-qhrx5 3a7f64b3-f2ad-4314-b6b4-2a6365fea608 up ocprhv-ptbf8-worker-1-8xv4w Provisioning 46m ocprhv-ptbf8-worker-1-chbls Deleting 47m ocprhv-ptbf8-worker-1-chbls 5da85f0c-8ed0-44f6-8b7d-0ba142a65c5d up ocprhv-ptbf8-worker-1-dqrth Deleting 47m ocprhv-ptbf8-worker-1-dqrth 40f2f5ec-c5b1-4316-abef-dbab8614da47 up ocprhv-ptbf8-worker-1-l52vf 69s ocprhv-ptbf8-worker-1-r4ddj 44s ocprhv-ptbf8-worker-1-xs6d9 Deleting 46m ocprhv-ptbf8-worker-1-xs6d9 62b66c93-389c-4bcf-aa2d-7f6689e11f72 up ~~~