Bug 1664591 - [cloud] Updating machineset "matchLabels" to a non-existent cluster results in lots of machines being created
Summary: [cloud] Updating machineset "matchLabels" to a non-existent cluster results i...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.1.0
Assignee: Jan Chaloupka
QA Contact: sunzhaohua
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-09 09:10 UTC by sunzhaohua
Modified: 2019-06-04 10:41 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:41:38 UTC
Target Upstream Version:


Attachments (Terms of Use)
controller manager logs (2.00 KB, text/plain)
2019-01-11 09:48 UTC, sunzhaohua
no flags Details
machine controller logs (7.69 KB, text/plain)
2019-01-11 09:49 UTC, sunzhaohua
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:41:43 UTC

Description sunzhaohua 2019-01-09 09:10:58 UTC
Description of problem:
Updating machineset "matchLabels" to a non-existent cluster results in lots of machines being created.

Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION                           AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.alpha-2019-01-09-005354   True        False         3h        Cluster version is 4.0.0-0.alpha-2019-01-09-005354

How reproducible:
Always

Steps to Reproduce:
1. Check cluster name
$ oc get cluster
NAME      AGE
zhsun     6h

2. Update machineset matchLabels to a non-existent cluster "zhsun-1"
$ oc get machineset zhsun-worker-us-east-2a-1 -o yaml
apiVersion: cluster.k8s.io/v1alpha1
kind: MachineSet
metadata:
  labels:
    sigs.k8s.io/cluster-api-cluster: zhsun
    sigs.k8s.io/cluster-api-machine-role: worker
    sigs.k8s.io/cluster-api-machine-type: worker
  name: zhsun-worker-us-east-2a-1
  namespace: openshift-cluster-api
spec:
  replicas: 1
  selector:
    matchLabels:
      sigs.k8s.io/cluster-api-cluster: zhsun-1
      sigs.k8s.io/cluster-api-machineset: zhsun-worker-us-east-2a-1
  template:
    metadata:
      creationTimestamp: null
      labels:
        sigs.k8s.io/cluster-api-cluster: zhsun
        sigs.k8s.io/cluster-api-machine-role: worker
        sigs.k8s.io/cluster-api-machine-type: worker
        sigs.k8s.io/cluster-api-machineset: zhsun-worker-us-east-2a-1
    spec:
      metadata:
        creationTimestamp: null
      providerConfig:
        value:
          ami:
            arn: null
            filters: null
            id: ami-093f9b728930dfd98
          apiVersion: awsproviderconfig.k8s.io/v1alpha1
          credentialsSecret: null
          deviceIndex: 0
          iamInstanceProfile:
            arn: null
            filters: null
            id: zhsun-worker-profile
          instanceType: m4.large
          keyName: null
          kind: AWSMachineProviderConfig
          loadBalancers: null
          metadata:
            creationTimestamp: null
          placement:
            availabilityZone: us-east-2a
            region: us-east-2
          publicIp: null
          securityGroups:
          - arn: null
            filters:
            - name: tag:Name
              values:
              - zhsun_worker_sg
            id: null
          subnet:
            arn: null
            filters:
            - name: tag:Name
              values:
              - zhsun-worker-us-east-2a
            id: null
          tags:
          - name: openshiftClusterID
            value: de2101bf-fd22-4a0e-85a5-a698eb5c98f5
          - name: kubernetes.io/cluster/zhsun
            value: owned
          userDataSecret:
            name: worker-user-data
      versions:
        kubelet: ""
status:
  observedGeneration: 2
  replicas: 0
  
  
3. $ oc get machine
NAME                              AGE
zhsun-master-0                    5h
zhsun-master-1                    5h
zhsun-master-2                    5h
zhsun-worker-us-east-2a-1-24jn7   2m
zhsun-worker-us-east-2a-1-26n8h   1m
zhsun-worker-us-east-2a-1-2dtpt   1m
zhsun-worker-us-east-2a-1-2gr5j   2m
zhsun-worker-us-east-2a-1-2l2td   1m
zhsun-worker-us-east-2a-1-2nmwr   23s
zhsun-worker-us-east-2a-1-2nsht   40s
zhsun-worker-us-east-2a-1-2qddj   58s
zhsun-worker-us-east-2a-1-2qtdd   14s
zhsun-worker-us-east-2a-1-2vhcf   23s
zhsun-worker-us-east-2a-1-2w2sw   55s
zhsun-worker-us-east-2a-1-2zhpd   35s
zhsun-worker-us-east-2a-1-442tw   46s
zhsun-worker-us-east-2a-1-46np4   2m
zhsun-worker-us-east-2a-1-47pcn   1m
zhsun-worker-us-east-2a-1-48gvv   52s
zhsun-worker-us-east-2a-1-4bzxd   2m
zhsun-worker-us-east-2a-1-4djcb   1m
zhsun-worker-us-east-2a-1-4dnl6   13s
zhsun-worker-us-east-2a-1-4f7fj   52s
zhsun-worker-us-east-2a-1-4gmr8   1m
zhsun-worker-us-east-2a-1-4grlg   2m
zhsun-worker-us-east-2a-1-4hdnq   2m
zhsun-worker-us-east-2a-1-4l4rg   1m
zhsun-worker-us-east-2a-1-4m9rl   2m
zhsun-worker-us-east-2a-1-4mxdb   2m
zhsun-worker-us-east-2a-1-4nb9q   55s
zhsun-worker-us-east-2a-1-4prk7   2m
zhsun-worker-us-east-2a-1-4pxs4   1m
zhsun-worker-us-east-2a-1-4q9xp   2m
zhsun-worker-us-east-2a-1-4rgjg   1m
zhsun-worker-us-east-2a-1-4rgz9   16s
zhsun-worker-us-east-2a-1-4rq7p   1m
zhsun-worker-us-east-2a-1-4rvq2   13s
zhsun-worker-us-east-2a-1-4rw6r   1m
zhsun-worker-us-east-2a-1-4sms4   1m
zhsun-worker-us-east-2a-1-4tkck   1m
zhsun-worker-us-east-2a-1-4vpqp   6s
zhsun-worker-us-east-2a-1-4w5p7   1m
zhsun-worker-us-east-2a-1-4xh8t   43s
zhsun-worker-us-east-2a-1-4zdxh   27s
zhsun-worker-us-east-2a-1-54brr   2m
zhsun-worker-us-east-2a-1-55m2n   2m
zhsun-worker-us-east-2a-1-56jwv   26s
zhsun-worker-us-east-2a-1-5fm7g   2m
zhsun-worker-us-east-2a-1-5g9m6   2m
zhsun-worker-us-east-2a-1-5lw6r   1m
zhsun-worker-us-east-2a-1-5nj2p   2m
zhsun-worker-us-east-2a-1-5nzr2   2m
zhsun-worker-us-east-2a-1-5prxt   45s
zhsun-worker-us-east-2a-1-5pttl   2m
zhsun-worker-us-east-2a-1-5qhcq   2m
zhsun-worker-us-east-2a-1-5qj7g   51s
zhsun-worker-us-east-2a-1-5rghb   1m
zhsun-worker-us-east-2a-1-5sklt   1m
zhsun-worker-us-east-2a-1-5wdhc   2m
zhsun-worker-us-east-2a-1-5xnsw   1m
zhsun-worker-us-east-2a-1-5zbw9   45s
zhsun-worker-us-east-2a-1-62tsr   2m
zhsun-worker-us-east-2a-1-64bks   2m
zhsun-worker-us-east-2a-1-65hv5   2m
zhsun-worker-us-east-2a-1-66426   53s
zhsun-worker-us-east-2a-1-68db7   1m
zhsun-worker-us-east-2a-1-69b96   28s
zhsun-worker-us-east-2a-1-69gc8   21s
zhsun-worker-us-east-2a-1-69tdp   2m
zhsun-worker-us-east-2a-1-6d6gr   2m
zhsun-worker-us-east-2a-1-6f6v4   1m
zhsun-worker-us-east-2a-1-6fkk4   33s
zhsun-worker-us-east-2a-1-6fl6g   14s
zhsun-worker-us-east-2a-1-6fm5q   4s
zhsun-worker-us-east-2a-1-6fmbf   2m
zhsun-worker-us-east-2a-1-6gvzk   2m
zhsun-worker-us-east-2a-1-6jbmc   2m
zhsun-worker-us-east-2a-1-6jbvv   1m
zhsun-worker-us-east-2a-1-6jhj6   44s

Actual results:
Lots of machines were created

Expected results:
No machines are created.

Additional info:

Comment 1 Jan Chaloupka 2019-01-10 08:17:57 UTC
Has any of the created machines corresponding aws instances provisioned? Can you share logs from the machine controller?

Comment 2 Jan Chaloupka 2019-01-10 15:20:15 UTC
Can you share logs from controller manager as well? The machineset controller will most likely keep complaining it does not have enough machines available.

Comment 3 sunzhaohua 2019-01-11 09:48:47 UTC
Created attachment 1519990 [details]
controller manager logs

Comment 4 sunzhaohua 2019-01-11 09:49:36 UTC
Created attachment 1519991 [details]
machine controller logs

Comment 5 sunzhaohua 2019-01-11 09:52:26 UTC
Yes, corresponding aws instances were provisioned. Attached is controller-manager and machine-controller logs.

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-9   True        False         19h       Cluster version is 4.0.0-9

$ oc get machine
NAME                              AGE
zhsun-master-0                    19h
zhsun-master-1                    19h
zhsun-master-2                    19h
zhsun-worker-us-east-2a-a-29qds   11s
zhsun-worker-us-east-2a-a-2lcmh   13s
zhsun-worker-us-east-2a-a-2rnsr   13s
zhsun-worker-us-east-2a-a-49kwk   13s
zhsun-worker-us-east-2a-a-4mkrx   1s
zhsun-worker-us-east-2a-a-4ppv7   5s
zhsun-worker-us-east-2a-a-4tc5g   2s
zhsun-worker-us-east-2a-a-5lbjn   4s
zhsun-worker-us-east-2a-a-6r99v   1s
zhsun-worker-us-east-2a-a-6rcn8   8s
zhsun-worker-us-east-2a-a-7584b   13s
zhsun-worker-us-east-2a-a-75dx2   2s
zhsun-worker-us-east-2a-a-7bk2c   3s
zhsun-worker-us-east-2a-a-7t4gc   12s
...


$ oc get node
NAME                                         STATUS     ROLES     AGE       VERSION
ip-10-0-12-68.us-east-2.compute.internal     Ready      master    19h       v1.11.0+406fc897d8
ip-10-0-128-108.us-east-2.compute.internal   Ready      worker    7m        v1.11.0+406fc897d8
ip-10-0-128-208.us-east-2.compute.internal   Ready      worker    6m        v1.11.0+406fc897d8
ip-10-0-128-212.us-east-2.compute.internal   Ready      worker    4m        v1.11.0+406fc897d8
ip-10-0-128-244.us-east-2.compute.internal   Ready      worker    3m        v1.11.0+406fc897d8
ip-10-0-128-58.us-east-2.compute.internal    Ready      worker    6m        v1.11.0+406fc897d8
ip-10-0-128-84.us-east-2.compute.internal    Ready      worker    5m        v1.11.0+406fc897d8
ip-10-0-129-116.us-east-2.compute.internal   Ready      worker    6m        v1.11.0+406fc897d8
ip-10-0-129-123.us-east-2.compute.internal   Ready      worker    5m        v1.11.0+406fc897d8
ip-10-0-129-140.us-east-2.compute.internal   Ready      worker    5m        v1.11.0+406fc897d8
ip-10-0-129-210.us-east-2.compute.internal   Ready      worker    6m        v1.11.0+406fc897d8
ip-10-0-129-221.us-east-2.compute.internal   Ready      worker    7m        v1.11.0+406fc897d8
ip-10-0-129-224.us-east-2.compute.internal   Ready      worker    7m        v1.11.0+406fc897d8
ip-10-0-129-40.us-east-2.compute.internal    Ready      worker    2m        v1.11.0+406fc897d8
ip-10-0-129-54.us-east-2.compute.internal    Ready      worker    4m        v1.11.0+406fc897d8
ip-10-0-129-70.us-east-2.compute.internal    Ready      worker    4m        v1.11.0+406fc897d8
ip-10-0-129-84.us-east-2.compute.internal    Ready      worker    17m       v1.11.0+406fc897d8
ip-10-0-129-94.us-east-2.compute.internal    Ready      worker    5m        v1.11.0+406fc897d8
ip-10-0-130-128.us-east-2.compute.internal   Ready      worker    5m        v1.11.0+406fc897d8
ip-10-0-130-173.us-east-2.compute.internal   Ready      worker    5m        v1.11.0+406fc897d8
ip-10-0-130-36.us-east-2.compute.internal    Ready      worker    6m        v1.11.0+406fc897d8
ip-10-0-131-120.us-east-2.compute.internal   Ready      worker    4m        v1.11.0+406fc897d8
ip-10-0-131-168.us-east-2.compute.internal   Ready      worker    4m        v1.11.0+406fc897d8
ip-10-0-131-169.us-east-2.compute.internal   Ready      worker    5m        v1.11.0+406fc897d8
ip-10-0-131-206.us-east-2.compute.internal   Ready      worker    4m        v1.11.0+406fc897d8
ip-10-0-131-238.us-east-2.compute.internal   Ready      worker    4m        v1.11.0+406fc897d8
....


$ oc get pod
NAME                                              READY     STATUS             RESTARTS   AGE
cluster-autoscaler-default-7c88c947bc-8ffvc       1/1       Running            0          2h
cluster-autoscaler-operator-6855f55d94-v8v6f      1/1       Running            0          20h
clusterapi-manager-controllers-785bcff647-rjhbw   3/4       CrashLoopBackOff   6          19h
machine-api-operator-67f964b4d-mn799              1/1       Running            0          20h

Comment 7 Jan Chaloupka 2019-01-15 15:13:42 UTC
Merged as downstream PR.

Comment 8 sunzhaohua 2019-01-18 06:51:09 UTC
Verified.

$ ./openshift-install version
./openshift-install v4.0.0-0.142.0.0-dirty
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.nightly-2019-01-17-185315   True        False         35m       Cluster version is 4.0.0-0.nightly-2019-01-17-185315

Update machineset matchLabels to a non-existent cluster, no machines were created.

$ oc logs -f clusterapi-manager-controllers-64584db79-d2hq5 -c controller-manager
E0118 06:28:07.595314       1 controller.go:156] "zhsun-worker-us-east-2a" machineset validation failed: spec.template.metadata.labels: Invalid value: map[string]string{"sigs.k8s.io/cluster-api-cluster":"zhsun", "sigs.k8s.io/cluster-api-machine-role":"worker", "sigs.k8s.io/cluster-api-machine-type":"worker", "sigs.k8s.io/cluster-api-machineset":"zhsun-worker-us-east-2a"}: `selector` does not match template `labels`
E0118 06:28:08.601097       1 controller.go:156] "zhsun-worker-us-east-2a" machineset validation failed: spec.template.metadata.labels: Invalid value: map[string]string{"sigs.k8s.io/cluster-api-machine-role":"worker", "sigs.k8s.io/cluster-api-machine-type":"worker", "sigs.k8s.io/cluster-api-machineset":"zhsun-worker-us-east-2a", "sigs.k8s.io/cluster-api-cluster":"zhsun"}: `selector` does not match template `labels`
E0118 06:28:09.601556       1 controller.go:156] "zhsun-worker-us-east-2a" machineset validation failed: spec.template.metadata.labels: Invalid value: map[string]string{"sigs.k8s.io/cluster-api-machine-type":"worker", "sigs.k8s.io/cluster-api-machineset":"zhsun-worker-us-east-2a", "sigs.k8s.io/cluster-api-cluster":"zhsun", "sigs.k8s.io/cluster-api-machine-role":"worker"}: `selector` does not match template `labels`

Comment 11 errata-xmlrpc 2019-06-04 10:41:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.