Hide Forgot
Description of problem: Updating machineset "matchLabels" to a non-existent cluster results in lots of machines being created. Version-Release number of selected component (if applicable): $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.alpha-2019-01-09-005354 True False 3h Cluster version is 4.0.0-0.alpha-2019-01-09-005354 How reproducible: Always Steps to Reproduce: 1. Check cluster name $ oc get cluster NAME AGE zhsun 6h 2. Update machineset matchLabels to a non-existent cluster "zhsun-1" $ oc get machineset zhsun-worker-us-east-2a-1 -o yaml apiVersion: cluster.k8s.io/v1alpha1 kind: MachineSet metadata: labels: sigs.k8s.io/cluster-api-cluster: zhsun sigs.k8s.io/cluster-api-machine-role: worker sigs.k8s.io/cluster-api-machine-type: worker name: zhsun-worker-us-east-2a-1 namespace: openshift-cluster-api spec: replicas: 1 selector: matchLabels: sigs.k8s.io/cluster-api-cluster: zhsun-1 sigs.k8s.io/cluster-api-machineset: zhsun-worker-us-east-2a-1 template: metadata: creationTimestamp: null labels: sigs.k8s.io/cluster-api-cluster: zhsun sigs.k8s.io/cluster-api-machine-role: worker sigs.k8s.io/cluster-api-machine-type: worker sigs.k8s.io/cluster-api-machineset: zhsun-worker-us-east-2a-1 spec: metadata: creationTimestamp: null providerConfig: value: ami: arn: null filters: null id: ami-093f9b728930dfd98 apiVersion: awsproviderconfig.k8s.io/v1alpha1 credentialsSecret: null deviceIndex: 0 iamInstanceProfile: arn: null filters: null id: zhsun-worker-profile instanceType: m4.large keyName: null kind: AWSMachineProviderConfig loadBalancers: null metadata: creationTimestamp: null placement: availabilityZone: us-east-2a region: us-east-2 publicIp: null securityGroups: - arn: null filters: - name: tag:Name values: - zhsun_worker_sg id: null subnet: arn: null filters: - name: tag:Name values: - zhsun-worker-us-east-2a id: null tags: - name: openshiftClusterID value: de2101bf-fd22-4a0e-85a5-a698eb5c98f5 - name: kubernetes.io/cluster/zhsun value: owned userDataSecret: name: worker-user-data versions: kubelet: "" status: observedGeneration: 2 replicas: 0 3. $ oc get machine NAME AGE zhsun-master-0 5h zhsun-master-1 5h zhsun-master-2 5h zhsun-worker-us-east-2a-1-24jn7 2m zhsun-worker-us-east-2a-1-26n8h 1m zhsun-worker-us-east-2a-1-2dtpt 1m zhsun-worker-us-east-2a-1-2gr5j 2m zhsun-worker-us-east-2a-1-2l2td 1m zhsun-worker-us-east-2a-1-2nmwr 23s zhsun-worker-us-east-2a-1-2nsht 40s zhsun-worker-us-east-2a-1-2qddj 58s zhsun-worker-us-east-2a-1-2qtdd 14s zhsun-worker-us-east-2a-1-2vhcf 23s zhsun-worker-us-east-2a-1-2w2sw 55s zhsun-worker-us-east-2a-1-2zhpd 35s zhsun-worker-us-east-2a-1-442tw 46s zhsun-worker-us-east-2a-1-46np4 2m zhsun-worker-us-east-2a-1-47pcn 1m zhsun-worker-us-east-2a-1-48gvv 52s zhsun-worker-us-east-2a-1-4bzxd 2m zhsun-worker-us-east-2a-1-4djcb 1m zhsun-worker-us-east-2a-1-4dnl6 13s zhsun-worker-us-east-2a-1-4f7fj 52s zhsun-worker-us-east-2a-1-4gmr8 1m zhsun-worker-us-east-2a-1-4grlg 2m zhsun-worker-us-east-2a-1-4hdnq 2m zhsun-worker-us-east-2a-1-4l4rg 1m zhsun-worker-us-east-2a-1-4m9rl 2m zhsun-worker-us-east-2a-1-4mxdb 2m zhsun-worker-us-east-2a-1-4nb9q 55s zhsun-worker-us-east-2a-1-4prk7 2m zhsun-worker-us-east-2a-1-4pxs4 1m zhsun-worker-us-east-2a-1-4q9xp 2m zhsun-worker-us-east-2a-1-4rgjg 1m zhsun-worker-us-east-2a-1-4rgz9 16s zhsun-worker-us-east-2a-1-4rq7p 1m zhsun-worker-us-east-2a-1-4rvq2 13s zhsun-worker-us-east-2a-1-4rw6r 1m zhsun-worker-us-east-2a-1-4sms4 1m zhsun-worker-us-east-2a-1-4tkck 1m zhsun-worker-us-east-2a-1-4vpqp 6s zhsun-worker-us-east-2a-1-4w5p7 1m zhsun-worker-us-east-2a-1-4xh8t 43s zhsun-worker-us-east-2a-1-4zdxh 27s zhsun-worker-us-east-2a-1-54brr 2m zhsun-worker-us-east-2a-1-55m2n 2m zhsun-worker-us-east-2a-1-56jwv 26s zhsun-worker-us-east-2a-1-5fm7g 2m zhsun-worker-us-east-2a-1-5g9m6 2m zhsun-worker-us-east-2a-1-5lw6r 1m zhsun-worker-us-east-2a-1-5nj2p 2m zhsun-worker-us-east-2a-1-5nzr2 2m zhsun-worker-us-east-2a-1-5prxt 45s zhsun-worker-us-east-2a-1-5pttl 2m zhsun-worker-us-east-2a-1-5qhcq 2m zhsun-worker-us-east-2a-1-5qj7g 51s zhsun-worker-us-east-2a-1-5rghb 1m zhsun-worker-us-east-2a-1-5sklt 1m zhsun-worker-us-east-2a-1-5wdhc 2m zhsun-worker-us-east-2a-1-5xnsw 1m zhsun-worker-us-east-2a-1-5zbw9 45s zhsun-worker-us-east-2a-1-62tsr 2m zhsun-worker-us-east-2a-1-64bks 2m zhsun-worker-us-east-2a-1-65hv5 2m zhsun-worker-us-east-2a-1-66426 53s zhsun-worker-us-east-2a-1-68db7 1m zhsun-worker-us-east-2a-1-69b96 28s zhsun-worker-us-east-2a-1-69gc8 21s zhsun-worker-us-east-2a-1-69tdp 2m zhsun-worker-us-east-2a-1-6d6gr 2m zhsun-worker-us-east-2a-1-6f6v4 1m zhsun-worker-us-east-2a-1-6fkk4 33s zhsun-worker-us-east-2a-1-6fl6g 14s zhsun-worker-us-east-2a-1-6fm5q 4s zhsun-worker-us-east-2a-1-6fmbf 2m zhsun-worker-us-east-2a-1-6gvzk 2m zhsun-worker-us-east-2a-1-6jbmc 2m zhsun-worker-us-east-2a-1-6jbvv 1m zhsun-worker-us-east-2a-1-6jhj6 44s Actual results: Lots of machines were created Expected results: No machines are created. Additional info:
Has any of the created machines corresponding aws instances provisioned? Can you share logs from the machine controller?
Can you share logs from controller manager as well? The machineset controller will most likely keep complaining it does not have enough machines available.
Created attachment 1519990 [details] controller manager logs
Created attachment 1519991 [details] machine controller logs
Yes, corresponding aws instances were provisioned. Attached is controller-manager and machine-controller logs. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-9 True False 19h Cluster version is 4.0.0-9 $ oc get machine NAME AGE zhsun-master-0 19h zhsun-master-1 19h zhsun-master-2 19h zhsun-worker-us-east-2a-a-29qds 11s zhsun-worker-us-east-2a-a-2lcmh 13s zhsun-worker-us-east-2a-a-2rnsr 13s zhsun-worker-us-east-2a-a-49kwk 13s zhsun-worker-us-east-2a-a-4mkrx 1s zhsun-worker-us-east-2a-a-4ppv7 5s zhsun-worker-us-east-2a-a-4tc5g 2s zhsun-worker-us-east-2a-a-5lbjn 4s zhsun-worker-us-east-2a-a-6r99v 1s zhsun-worker-us-east-2a-a-6rcn8 8s zhsun-worker-us-east-2a-a-7584b 13s zhsun-worker-us-east-2a-a-75dx2 2s zhsun-worker-us-east-2a-a-7bk2c 3s zhsun-worker-us-east-2a-a-7t4gc 12s ... $ oc get node NAME STATUS ROLES AGE VERSION ip-10-0-12-68.us-east-2.compute.internal Ready master 19h v1.11.0+406fc897d8 ip-10-0-128-108.us-east-2.compute.internal Ready worker 7m v1.11.0+406fc897d8 ip-10-0-128-208.us-east-2.compute.internal Ready worker 6m v1.11.0+406fc897d8 ip-10-0-128-212.us-east-2.compute.internal Ready worker 4m v1.11.0+406fc897d8 ip-10-0-128-244.us-east-2.compute.internal Ready worker 3m v1.11.0+406fc897d8 ip-10-0-128-58.us-east-2.compute.internal Ready worker 6m v1.11.0+406fc897d8 ip-10-0-128-84.us-east-2.compute.internal Ready worker 5m v1.11.0+406fc897d8 ip-10-0-129-116.us-east-2.compute.internal Ready worker 6m v1.11.0+406fc897d8 ip-10-0-129-123.us-east-2.compute.internal Ready worker 5m v1.11.0+406fc897d8 ip-10-0-129-140.us-east-2.compute.internal Ready worker 5m v1.11.0+406fc897d8 ip-10-0-129-210.us-east-2.compute.internal Ready worker 6m v1.11.0+406fc897d8 ip-10-0-129-221.us-east-2.compute.internal Ready worker 7m v1.11.0+406fc897d8 ip-10-0-129-224.us-east-2.compute.internal Ready worker 7m v1.11.0+406fc897d8 ip-10-0-129-40.us-east-2.compute.internal Ready worker 2m v1.11.0+406fc897d8 ip-10-0-129-54.us-east-2.compute.internal Ready worker 4m v1.11.0+406fc897d8 ip-10-0-129-70.us-east-2.compute.internal Ready worker 4m v1.11.0+406fc897d8 ip-10-0-129-84.us-east-2.compute.internal Ready worker 17m v1.11.0+406fc897d8 ip-10-0-129-94.us-east-2.compute.internal Ready worker 5m v1.11.0+406fc897d8 ip-10-0-130-128.us-east-2.compute.internal Ready worker 5m v1.11.0+406fc897d8 ip-10-0-130-173.us-east-2.compute.internal Ready worker 5m v1.11.0+406fc897d8 ip-10-0-130-36.us-east-2.compute.internal Ready worker 6m v1.11.0+406fc897d8 ip-10-0-131-120.us-east-2.compute.internal Ready worker 4m v1.11.0+406fc897d8 ip-10-0-131-168.us-east-2.compute.internal Ready worker 4m v1.11.0+406fc897d8 ip-10-0-131-169.us-east-2.compute.internal Ready worker 5m v1.11.0+406fc897d8 ip-10-0-131-206.us-east-2.compute.internal Ready worker 4m v1.11.0+406fc897d8 ip-10-0-131-238.us-east-2.compute.internal Ready worker 4m v1.11.0+406fc897d8 .... $ oc get pod NAME READY STATUS RESTARTS AGE cluster-autoscaler-default-7c88c947bc-8ffvc 1/1 Running 0 2h cluster-autoscaler-operator-6855f55d94-v8v6f 1/1 Running 0 20h clusterapi-manager-controllers-785bcff647-rjhbw 3/4 CrashLoopBackOff 6 19h machine-api-operator-67f964b4d-mn799 1/1 Running 0 20h
Upstream PR: https://github.com/kubernetes-sigs/cluster-api/pull/669 Downstream PR: https://github.com/openshift/cluster-api-provider-aws/pull/140
Merged as downstream PR.
Verified. $ ./openshift-install version ./openshift-install v4.0.0-0.142.0.0-dirty $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-01-17-185315 True False 35m Cluster version is 4.0.0-0.nightly-2019-01-17-185315 Update machineset matchLabels to a non-existent cluster, no machines were created. $ oc logs -f clusterapi-manager-controllers-64584db79-d2hq5 -c controller-manager E0118 06:28:07.595314 1 controller.go:156] "zhsun-worker-us-east-2a" machineset validation failed: spec.template.metadata.labels: Invalid value: map[string]string{"sigs.k8s.io/cluster-api-cluster":"zhsun", "sigs.k8s.io/cluster-api-machine-role":"worker", "sigs.k8s.io/cluster-api-machine-type":"worker", "sigs.k8s.io/cluster-api-machineset":"zhsun-worker-us-east-2a"}: `selector` does not match template `labels` E0118 06:28:08.601097 1 controller.go:156] "zhsun-worker-us-east-2a" machineset validation failed: spec.template.metadata.labels: Invalid value: map[string]string{"sigs.k8s.io/cluster-api-machine-role":"worker", "sigs.k8s.io/cluster-api-machine-type":"worker", "sigs.k8s.io/cluster-api-machineset":"zhsun-worker-us-east-2a", "sigs.k8s.io/cluster-api-cluster":"zhsun"}: `selector` does not match template `labels` E0118 06:28:09.601556 1 controller.go:156] "zhsun-worker-us-east-2a" machineset validation failed: spec.template.metadata.labels: Invalid value: map[string]string{"sigs.k8s.io/cluster-api-machine-type":"worker", "sigs.k8s.io/cluster-api-machineset":"zhsun-worker-us-east-2a", "sigs.k8s.io/cluster-api-cluster":"zhsun", "sigs.k8s.io/cluster-api-machine-role":"worker"}: `selector` does not match template `labels`
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758