Bug 1833322

That is not a bug, in vSphere we don't have instance types and regions/zone are optional fields.

Can you validate that zones/regions are set when they are present?

Created a machineset with zone as "Test" and Region as "A" , I am not sure if it is the correct way to add zones or region or not , but it seems the provider will simply ignore if its invalid

NAME                               PHASE         TYPE   REGION   ZONE   AGE
miyadav-11-rdnr9-master-0          Running                              109m
miyadav-11-rdnr9-master-1          Running                              109m
miyadav-11-rdnr9-master-2          Running                              109m
miyadav-11-rdnr9-worker-4gqcp      Running                              101m
miyadav-11-rdnr9-worker-rz-pslsl   Provisioned                          103s
miyadav-11-rdnr9-worker-vzds4      Running                              101m

[miyadav@miyadav bugaws]$ oc get machines miyadav-11-rdnr9-worker-rz-pslsl -o yaml | grep -i zone
    machine.openshift.io/zone: ""
            f:zone: {}
          f:machine.openshift.io/zone: {}
      zone: Test

[miyadav@miyadav ]$ oc get machines
NAME                               PHASE         TYPE   REGION   ZONE   AGE
miyadav-11-rdnr9-master-0          Running                              110m
miyadav-11-rdnr9-master-1          Running                              110m
miyadav-11-rdnr9-master-2          Running                              110m
miyadav-11-rdnr9-worker-4gqcp      Running                              101m
miyadav-11-rdnr9-worker-rz-pslsl   Provisioned                          2m25s
miyadav-11-rdnr9-worker-vzds4      Running                              101m

[miyadav@miyadav ]$ oc get machines
NAME                               PHASE     TYPE   REGION   ZONE   AGE
miyadav-11-rdnr9-master-0          Running                          115m
miyadav-11-rdnr9-master-1          Running                          115m
miyadav-11-rdnr9-master-2          Running                          115m
miyadav-11-rdnr9-worker-4gqcp      Running                          107m
miyadav-11-rdnr9-worker-rz-pslsl   Running                          8m8s
miyadav-11-rdnr9-worker-vzds4      Running                          107m

[miyadav@miyadav bugaws]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-05-12-224129   True        False         96m     Cluster version is 4.5.0-0.nightly-2020-05-12-224129



Additional logs :
.
.
.
:<nil> IpConfig:0xc00053bda0 NetBIOSConfig:<nil>}
I0513 04:59:30.898977       1 reconciler.go:839] Getting network status: getting guest info: network: {DynamicData:{} Network:VM Network IpAddress:[136.144.52.235 fe80::e64f:a5a5:a478:475d] MacAddress:00:50:56:ab:ee:0b Connected:true DeviceConfigId:4000 DnsConfig:<nil> IpConfig:0xc00053bdd0 NetBIOSConfig:<nil>}
I0513 04:59:30.898988       1 reconciler.go:839] Getting network status: getting guest info: network: {DynamicData:{} Network: IpAddress:[10.129.2.1 fe80::808d:22ff:fedc:d381] MacAddress:82:8d:22:dc:d3:81 Connected:true DeviceConfigId:-1 DnsConfig:<nil> IpConfig:0xc00053be00 NetBIOSConfig:<nil>}
I0513 04:59:30.898995       1 reconciler.go:839] Getting network status: getting guest info: network: {DynamicData:{} Network: IpAddress:[] MacAddress:66:e7:1c:44:eb:b2 Connected:true DeviceConfigId:-1 DnsConfig:<nil> IpConfig:0xc00053be30 NetBIOSConfig:<nil>}
I0513 04:59:30.899002       1 reconciler.go:839] Getting network status: getting guest info: network: {DynamicData:{} Network: IpAddress:[fe80::b4cd:48ff:fe26:2891] MacAddress:b6:cd:48:26:28:91 Connected:true DeviceConfigId:-1 DnsConfig:<nil> IpConfig:0xc00053be60 NetBIOSConfig:<nil>}
I0513 04:59:30.900937       1 reconciler.go:324] miyadav-11-rdnr9-worker-rz-pslsl: reconciling network: IP addresses: [{InternalIP 136.144.52.235} {InternalIP fe80::e64f:a5a5:a478:475d} {InternalDNS miyadav-11-rdnr9-worker-rz-pslsl}]
I0513 04:59:30.900961       1 reconciler.go:620] miyadav-11-rdnr9-worker-rz-pslsl: Updating provider status
I0513 04:59:30.906255       1 machine_scope.go:99] miyadav-11-rdnr9-worker-rz-pslsl: patching machine

Tried below as well , but did not get zones/region  updated .

oc edit cm cloud-provider-config -n openshift-config 
..
 [Labels]
    region = k8s-region
    zone = k8s-zone
.
.

Scaled machineset -

 oc scale machineset qe-miyadav-zv8wm-worker-new --replicas=4

[miyadav@miyadav ManualRun]$ oc get machines
NAME                                PHASE         TYPE   REGION   ZONE   AGE
qe-miyadav-zv8wm-master-0           Running                              172m
qe-miyadav-zv8wm-master-1           Running                              172m
qe-miyadav-zv8wm-master-2           Running                              172m
qe-miyadav-zv8wm-worker-jnw7x       Running                              162m
qe-miyadav-zv8wm-worker-lf7gl       Running                              27m
qe-miyadav-zv8wm-worker-mw5cj       Running                              62m
qe-miyadav-zv8wm-worker-new-2nksz   Running                              38m
qe-miyadav-zv8wm-worker-new-ff8gt   Provisioned                          7m58s
qe-miyadav-zv8wm-worker-new-g297p   Provisioned                          2m28s
qe-miyadav-zv8wm-worker-new-mhq8h   Provisioned                          7m58s

but REGION and Zone were not populated.

Have a look ..
Jira - https://issues.redhat.com/browse/OCPCLOUD-751 , providing this for tracking purpose.

Hi,
In order for region/zone to appear, you should also attach tags to VM on vSphere: tag category - "k8s-region" and tag name - "any-name", same for k8s-zone

Can you verify that controller works as expected if tags are assigned to VM?

While trying to add tag , I am getting this events in logs  and tags are not attached to VMs it seems ..
171m        Normal    Update                  machine/qe-miyadav-zv8wm-worker-new-mhq8h   Updated Machine qe-miyadav-zv8wm-worker-new-mhq8h
172m        Normal    DetectedUnhealthy       machine/qe-miyadav-zv8wm-worker-new-mhq8h   Machine openshift-machine-api/mhc1/qe-miyadav-zv8wm-worker-new-mhq8h/qe-miyadav-zv8wm-worker-new-mhq8h has unhealthy node qe-miyadav-zv8wm-worker-new-mhq8h
150m        Normal    Deleted                 machine/qe-miyadav-zv8wm-worker-new-mhq8h   Node "qe-miyadav-zv8wm-worker-new-mhq8h" drained
150m        Warning   FailedDelete            machine/qe-miyadav-zv8wm-worker-new-mhq8h   destroying vm in progress, reconciling
150m        Warning   FailedDelete            machine/qe-miyadav-zv8wm-worker-new-mhq8h   task task-1939 has not finished
150m        Normal    Delete                  machine/qe-miyadav-zv8wm-worker-new-mhq8h   Deleted machine qe-miyadav-zv8wm-worker-new-mhq8h
171m        Normal    DetectedUnhealthy       machine/qe-miyadav-zv8wm-worker-new-wk7rd   Machine openshift-machine-api/mhc1/qe-miyadav-zv8wm-worker-new-wk7rd/ has unhealthy node
171m        Normal    Create                  machine/qe-miyadav-zv8wm-worker-new-wk7rd   Created Machine qe-miyadav-zv8wm-worker-new-wk7rd
171m        Warning   FailedCreate            machine/qe-miyadav-zv8wm-worker-new-wk7rd   task task-1921 has not finished
140m        Normal    Update                  machine/qe-miyadav-zv8wm-worker-new-wk7rd   Updated Machine qe-miyadav-zv8wm-worker-new-wk7rd
3m17s       Normal    DetectedUnhealthy       machine/qe-miyadav-zv8wm-worker-rxjh4       Machine openshift-machine-api/mhc1/qe-miyadav-zv8wm-worker-rxjh4/ has unhealthy node
6m33s       Normal    Create                  machine/qe-miyadav-zv8wm-worker-rxjh4       Created Machine qe-miyadav-zv8wm-worker-rxjh4
6m24s       Warning   FailedCreate            machine/qe-miyadav-zv8wm-worker-rxjh4       task task-2082 has not finished
6m15s       Warning   FailedCreate            machine/qe-miyadav-zv8wm-worker-rxjh4       failed to reconcile tags: get category urn:vmomi:InventoryServiceTag:82d61840-9c56-428b-bfef-f898f55dfb3b:GLOBAL failed for Get https://vcsa2-qe.vmware.devcluster.openshift.com/rest/com/vmware/cis/tagging/tag/id:urn:vmomi:InventoryServiceTag:82d61840-9c56-428b-bfef-f898f55dfb3b:GLOBAL: dial tcp: lookup vcsa2-qe.vmware.devcluster.openshift.com on 172.30.0.10:53: server misbehaving

There is an issue with vCenter, can you try to verify it again?

Did it multiple times , it did updated tag zone ..

[miyadav@miyadav bugvsphere]$ oc logs machine-api-controllers-b87757dfb-jgnjw -c machine-controller | grep "1833322"
I0527 03:25:13.782380       1 reconciler.go:749] VirtualMachine:vm-683 has zone tag (k8s-zone) with value bug1833322
I0527 03:25:18.887215       1 reconciler.go:749] VirtualMachine:vm-683 has zone tag (k8s-zone) with value bug1833322
I0527 03:27:45.057047       1 reconciler.go:749] VirtualMachine:vm-683 has zone tag (k8s-zone) with value bug1833322
I0527 03:27:47.560759       1 reconciler.go:749] VirtualMachine:vm-683 has zone tag (k8s-zone) with value bug1833322


[miyadav@miyadav bugvsphere]$ oc get machines
NAME                                 PHASE     TYPE   REGION   ZONE         AGE
miyadav-vsph-t2fxq-master-0          Running                                64m
miyadav-vsph-t2fxq-master-1          Running                                64m
miyadav-vsph-t2fxq-master-2          Running                                64m
miyadav-vsph-t2fxq-worker-l4hm9      Running                   bug1833322   56m
miyadav-vsph-t2fxq-worker-pjw4r      Running                                56m
miyadav-vsph-t2fxq-worker-t-556rf    Running                                4m30s
miyadav-vsph-t2fxq-worker-tg-bwcx4   Running                                13m
miyadav-vsph-t2fxq-worker-tg-jssk9   Running                                6m17s
miyadav-vsph-t2fxq-worker-tg-nj254   Running                                13m

but when I tried multiple times , I couldnt get it always , nothing in logs that I can see , like error etc. 

not sure what else is needed , it is not happening always..

Please let me know , what else we can try to figure out , why not happening always .. ( cm cloud-provider-config values k8s-zone and k8s-region are the category for tags in vpshere)

For clarification: do all VMs in vSphere have proper tags attached?

Created attachment 1692922 [details]
image

Adding more info for comment#1 , One of tags is attached by installer , rest we added manually . , seems they do get attached . on VMs

This slipped 4.5. tagging with upcomingSprint

I double-checked everything, all works as expected in my environment. If zones and regions are missing on machines after tags were added to VMs,  you need to wait for the controller to update machines or restart the controllers pod.

Created attachment 1695974 [details]
vsphereUI_tags_added_confirmation

this is just to refer in case someone else reviews the bug later

Thanks Alexander for your help , could consistently see the region and tags appearing , just that it takes time for reflecting as you said.

[miyadav@miyadav ~]$ oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-rc.1   True        False         59m     Cluster version is 4.5.0-rc.1


Steps:
Updated cm cloud-provider-config like below - 

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  config: |
    [Global]
    secret-name = "vsphere-creds"
    secret-namespace = "kube-system"
    insecure-flag = "1"

    [Labels]
    region = k8s-region
    zone = k8s-zone

    [Workspace]
    server = "vcsa2-qe.vmware.devcluster.openshift.com"
    datacenter = "dc1"
    default-datastore = "10TB-GOLD"
    folder = "/dc1/vm/jhou-tx96r" [VirtualCenter "vcsa2-qe.vmware.devcluster.openshift.com"]
    datacenters = "dc1"
kind: ConfigMap
metadata:
  creationTimestamp: "2020-06-08T03:19:50Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data: {}
    manager: cluster-bootstrap
    operation: Update
    time: "2020-06-08T03:19:50Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:config: {}
    manager: oc
    operation: Update
    time: "2020-06-08T04:08:59Z"
  name: cloud-provider-config
  namespace: openshift-config
  resourceVersion: "32276"
  selfLink: /api/v1/namespaces/openshift-config/configmaps/cloud-provider-config
  uid: bb9dd6f5-6149-4152-8ca9-de6982e54bc6
~                                                                                                                                                                                                                  
~                                                            

2.Added tags as shown in snap (attachement)

3.Zones and region attached 
[miyadav@miyadav ~]$ oc get machines
NAME                      PHASE         TYPE   REGION          ZONE            AGE
jhou-tx96r-master-0       Running                                              69m
jhou-tx96r-master-1       Running                                              69m
jhou-tx96r-master-2       Running                                              69m
jhou-tx96r-worker-6jj8r   Provisioned                                          2m54s
jhou-tx96r-worker-gbs54   Running              bug1833322_v1   test-tag-zone   60m
jhou-tx96r-worker-pv64z   Running                                              60m


Additional info
Even after deleting tags and reverting configmap config , the region and zone exists.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196