Bug 2087177

Just to summarize an offlist discussion: It seems that VMIs have an instance id. Whenever this instance ID is changing, then cloud init assumes that the VM is a new instance.
It could be that managed VMI - such as VMIs tied to VMs - should maintain it's instance ID throughout the VM life-cycle. IOWthe VM might need to provide the instance ID for the VMIs.

@Damien, What was the RHEL OS Version of the VM that was created from the Template?

Please also provide the VM yaml

[cloud-user@rhel8-unknown-hornet ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.6 (Ootpa)

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
    kubemacpool.io/transaction-timestamp: '2022-05-17T13:53:55.894724909Z'
    kubevirt.io/latest-observed-api-version: v1
    kubevirt.io/storage-observed-api-version: v1alpha3
    name.os.template.kubevirt.io/rhel8.5: Red Hat Enterprise Linux 8.0 or higher
    vm.kubevirt.io/validations: |
      [
        {
          "name": "minimal-required-memory",
          "path": "jsonpath::.spec.domain.resources.requests.memory",
          "rule": "integer",
          "message": "This VM requires more memory.",
          "min": 1610612736
        }
      ]
  resourceVersion: '7477775'
  name: rhel8-unknown-hornet
  uid: a60525b4-6ec5-4389-86ef-8fc8b5244e53
  creationTimestamp: '2022-05-11T21:10:18Z'
  generation: 3
  managedFields:
    - apiVersion: kubevirt.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:name.os.template.kubevirt.io/rhel8.5': {}
            'f:vm.kubevirt.io/validations': {}
          'f:labels':
            'f:vm.kubevirt.io/template.version': {}
            'f:vm.kubevirt.io/template.namespace': {}
            'f:os.template.kubevirt.io/rhel8.5': {}
            'f:app': {}
            .: {}
            'f:vm.kubevirt.io/template.revision': {}
            'f:workload.template.kubevirt.io/server': {}
            'f:flavor.template.kubevirt.io/small': {}
            'f:vm.kubevirt.io/template': {}
        'f:spec':
          .: {}
          'f:dataVolumeTemplates': {}
          'f:template':
            .: {}
            'f:metadata':
              .: {}
              'f:annotations':
                .: {}
                'f:vm.kubevirt.io/flavor': {}
                'f:vm.kubevirt.io/os': {}
                'f:vm.kubevirt.io/workload': {}
              'f:labels':
                .: {}
                'f:flavor.template.kubevirt.io/small': {}
                'f:kubevirt.io/domain': {}
                'f:kubevirt.io/size': {}
                'f:os.template.kubevirt.io/rhel8.5': {}
                'f:vm.kubevirt.io/name': {}
                'f:workload.template.kubevirt.io/server': {}
            'f:spec':
              .: {}
              'f:accessCredentials': {}
              'f:domain':
                .: {}
                'f:cpu':
                  .: {}
                  'f:cores': {}
                  'f:sockets': {}
                  'f:threads': {}
                'f:devices':
                  .: {}
                  'f:disks': {}
                  'f:interfaces': {}
                  'f:networkInterfaceMultiqueue': {}
                  'f:rng': {}
                'f:machine':
                  .: {}
                  'f:type': {}
                'f:resources':
                  .: {}
                  'f:requests':
                    .: {}
                    'f:memory': {}
              'f:evictionStrategy': {}
              'f:hostname': {}
              'f:networks': {}
              'f:terminationGracePeriodSeconds': {}
              'f:volumes': {}
      manager: Mozilla
      operation: Update
      time: '2022-05-11T21:10:18Z'
    - apiVersion: kubevirt.io/v1alpha3
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:conditions': {}
          'f:created': {}
          'f:printableStatus': {}
          'f:ready': {}
      manager: Go-http-client
      operation: Update
      subresource: status
      time: '2022-05-17T13:54:00Z'
  namespace: damiens-vms
  labels:
    app: rhel8-unknown-hornet
    flavor.template.kubevirt.io/small: 'true'
    os.template.kubevirt.io/rhel8.5: 'true'
    vm.kubevirt.io/template: rhel8-server-small
    vm.kubevirt.io/template.namespace: openshift
    vm.kubevirt.io/template.revision: '1'
    vm.kubevirt.io/template.version: v0.19.3
    workload.template.kubevirt.io/server: 'true'
spec:
  dataVolumeTemplates:
    - apiVersion: cdi.kubevirt.io/v1beta1
      kind: DataVolume
      metadata:
        creationTimestamp: null
        name: rhel8-unknown-hornet
      spec:
        source:
          pvc:
            name: rhel8-1f05497b8847
            namespace: openshift-virtualization-os-images
        storage:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: '11362347344'
          storageClassName: sno-storage
          volumeMode: Filesystem
  running: true
  template:
    metadata:
      annotations:
        vm.kubevirt.io/flavor: small
        vm.kubevirt.io/os: rhel8
        vm.kubevirt.io/workload: server
      creationTimestamp: null
      labels:
        flavor.template.kubevirt.io/small: 'true'
        kubevirt.io/domain: rhel8-unknown-hornet
        kubevirt.io/size: small
        os.template.kubevirt.io/rhel8.5: 'true'
        vm.kubevirt.io/name: rhel8-unknown-hornet
        workload.template.kubevirt.io/server: 'true'
    spec:
      accessCredentials:
        - sshPublicKey:
            propagationMethod:
              configDrive: {}
            source:
              secret:
                secretName: authorizedsshkeys-rhel8-unknown-hornet
      domain:
        cpu:
          cores: 1
          sockets: 1
          threads: 1
        devices:
          disks:
            - bootOrder: 1
              disk:
                bus: virtio
              name: rhel8-unknown-hornet
            - disk:
                bus: virtio
              name: cloudinitdisk
          interfaces:
            - macAddress: '02:5e:0f:00:00:02'
              masquerade: {}
              name: default
          networkInterfaceMultiqueue: true
          rng: {}
        machine:
          type: pc-q35-rhel8.4.0
        resources:
          requests:
            memory: 2Gi
      evictionStrategy: LiveMigrate
      hostname: rhel8-unknown-hornet
      networks:
        - name: default
          pod: {}
      terminationGracePeriodSeconds: 180
      volumes:
        - dataVolume:
            name: rhel8-unknown-hornet
          name: rhel8-unknown-hornet
        - cloudInitConfigDrive:
            userData: |-
              #cloud-config
              user: cloud-user
              password: vgwo-gnbm-xrq3
              chpasswd: { expire: False }
          name: cloudinitdisk
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2022-05-17T13:53:58Z'
      status: 'True'
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: null
      message: >-
        cannot migrate VMI: PVC rhel8-unknown-hornet is not shared, live
        migration requires that all PVCs must be shared (using ReadWriteMany
        access mode)
      reason: DisksNotLiveMigratable
      status: 'False'
      type: LiveMigratable
    - lastProbeTime: '2022-05-17T13:59:05Z'
      lastTransitionTime: null
      status: 'True'
      type: AgentConnected
  created: true
  printableStatus: Running
  ready: true
  volumeSnapshotStatuses:
    - enabled: false
      name: rhel8-unknown-hornet
      reason: >-
        No VolumeSnapshotClass: Volume snapshots are not configured for this
        StorageClass [sno-storage] [rhel8-unknown-hornet]
    - enabled: false
      name: cloudinitdisk
      reason: 'Snapshot is not supported for this volumeSource type [cloudinitdisk]'

Posted https://github.com/kubevirt/kubevirt/pull/7807 to fix this.

Verified on CNV v4.11.1-3, SSH key still the same after restarting VM

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.11.1 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8750