Description of problem: If the pod hosting a VM is killed, when the pod is recreated and the VM restarted, the SSH keys have been changed thereby triggering a known_hosts mismatch error to a user logging in. Version-Release number of selected component (if applicable): OCP 4.10.12 (SNO); CNV 4.10.0 How reproducible: Reliably Steps to Reproduce: 1. Create RHEL VM from template 2. Log in to VM and accept host key... note creation dates of host keys in `/etc/ssh` 3. Stop and Start (or Restart) the VM from the Actions drop-down on the VM Details screen. 4. Log in and see known_hosts error 5. Delete known_hosts key and log in and see SSH keys with later creation dates Actual results: SSH keys are regenerated when VM is restarted from the UI and known_hosts error results. This could potentially result in increased concern from the owner of a VM that crashes unexpectedly. Expected results: After VM reboots, user can log in the same SSH keys still exist. Additional info: If the VM is rebooted from "inside the VM" with `reboot` command, this problem does not exist. Only when restarted from the OCP console. It appears to be because restarting from the OCP console recreates the pod which changes the instance_id in cloud-init which triggers cloud-init to re-run its init processes.
Just to summarize an offlist discussion: It seems that VMIs have an instance id. Whenever this instance ID is changing, then cloud init assumes that the VM is a new instance. It could be that managed VMI - such as VMIs tied to VMs - should maintain it's instance ID throughout the VM life-cycle. IOWthe VM might need to provide the instance ID for the VMIs.
@Damien, What was the RHEL OS Version of the VM that was created from the Template?
Please also provide the VM yaml
[cloud-user@rhel8-unknown-hornet ~]$ cat /etc/redhat-release Red Hat Enterprise Linux release 8.6 (Ootpa)
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: annotations: kubemacpool.io/transaction-timestamp: '2022-05-17T13:53:55.894724909Z' kubevirt.io/latest-observed-api-version: v1 kubevirt.io/storage-observed-api-version: v1alpha3 name.os.template.kubevirt.io/rhel8.5: Red Hat Enterprise Linux 8.0 or higher vm.kubevirt.io/validations: | [ { "name": "minimal-required-memory", "path": "jsonpath::.spec.domain.resources.requests.memory", "rule": "integer", "message": "This VM requires more memory.", "min": 1610612736 } ] resourceVersion: '7477775' name: rhel8-unknown-hornet uid: a60525b4-6ec5-4389-86ef-8fc8b5244e53 creationTimestamp: '2022-05-11T21:10:18Z' generation: 3 managedFields: - apiVersion: kubevirt.io/v1 fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:annotations': .: {} 'f:name.os.template.kubevirt.io/rhel8.5': {} 'f:vm.kubevirt.io/validations': {} 'f:labels': 'f:vm.kubevirt.io/template.version': {} 'f:vm.kubevirt.io/template.namespace': {} 'f:os.template.kubevirt.io/rhel8.5': {} 'f:app': {} .: {} 'f:vm.kubevirt.io/template.revision': {} 'f:workload.template.kubevirt.io/server': {} 'f:flavor.template.kubevirt.io/small': {} 'f:vm.kubevirt.io/template': {} 'f:spec': .: {} 'f:dataVolumeTemplates': {} 'f:template': .: {} 'f:metadata': .: {} 'f:annotations': .: {} 'f:vm.kubevirt.io/flavor': {} 'f:vm.kubevirt.io/os': {} 'f:vm.kubevirt.io/workload': {} 'f:labels': .: {} 'f:flavor.template.kubevirt.io/small': {} 'f:kubevirt.io/domain': {} 'f:kubevirt.io/size': {} 'f:os.template.kubevirt.io/rhel8.5': {} 'f:vm.kubevirt.io/name': {} 'f:workload.template.kubevirt.io/server': {} 'f:spec': .: {} 'f:accessCredentials': {} 'f:domain': .: {} 'f:cpu': .: {} 'f:cores': {} 'f:sockets': {} 'f:threads': {} 'f:devices': .: {} 'f:disks': {} 'f:interfaces': {} 'f:networkInterfaceMultiqueue': {} 'f:rng': {} 'f:machine': .: {} 'f:type': {} 'f:resources': .: {} 'f:requests': .: {} 'f:memory': {} 'f:evictionStrategy': {} 'f:hostname': {} 'f:networks': {} 'f:terminationGracePeriodSeconds': {} 'f:volumes': {} manager: Mozilla operation: Update time: '2022-05-11T21:10:18Z' - apiVersion: kubevirt.io/v1alpha3 fieldsType: FieldsV1 fieldsV1: 'f:status': 'f:conditions': {} 'f:created': {} 'f:printableStatus': {} 'f:ready': {} manager: Go-http-client operation: Update subresource: status time: '2022-05-17T13:54:00Z' namespace: damiens-vms labels: app: rhel8-unknown-hornet flavor.template.kubevirt.io/small: 'true' os.template.kubevirt.io/rhel8.5: 'true' vm.kubevirt.io/template: rhel8-server-small vm.kubevirt.io/template.namespace: openshift vm.kubevirt.io/template.revision: '1' vm.kubevirt.io/template.version: v0.19.3 workload.template.kubevirt.io/server: 'true' spec: dataVolumeTemplates: - apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: creationTimestamp: null name: rhel8-unknown-hornet spec: source: pvc: name: rhel8-1f05497b8847 namespace: openshift-virtualization-os-images storage: accessModes: - ReadWriteOnce resources: requests: storage: '11362347344' storageClassName: sno-storage volumeMode: Filesystem running: true template: metadata: annotations: vm.kubevirt.io/flavor: small vm.kubevirt.io/os: rhel8 vm.kubevirt.io/workload: server creationTimestamp: null labels: flavor.template.kubevirt.io/small: 'true' kubevirt.io/domain: rhel8-unknown-hornet kubevirt.io/size: small os.template.kubevirt.io/rhel8.5: 'true' vm.kubevirt.io/name: rhel8-unknown-hornet workload.template.kubevirt.io/server: 'true' spec: accessCredentials: - sshPublicKey: propagationMethod: configDrive: {} source: secret: secretName: authorizedsshkeys-rhel8-unknown-hornet domain: cpu: cores: 1 sockets: 1 threads: 1 devices: disks: - bootOrder: 1 disk: bus: virtio name: rhel8-unknown-hornet - disk: bus: virtio name: cloudinitdisk interfaces: - macAddress: '02:5e:0f:00:00:02' masquerade: {} name: default networkInterfaceMultiqueue: true rng: {} machine: type: pc-q35-rhel8.4.0 resources: requests: memory: 2Gi evictionStrategy: LiveMigrate hostname: rhel8-unknown-hornet networks: - name: default pod: {} terminationGracePeriodSeconds: 180 volumes: - dataVolume: name: rhel8-unknown-hornet name: rhel8-unknown-hornet - cloudInitConfigDrive: userData: |- #cloud-config user: cloud-user password: vgwo-gnbm-xrq3 chpasswd: { expire: False } name: cloudinitdisk status: conditions: - lastProbeTime: null lastTransitionTime: '2022-05-17T13:53:58Z' status: 'True' type: Ready - lastProbeTime: null lastTransitionTime: null message: >- cannot migrate VMI: PVC rhel8-unknown-hornet is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode) reason: DisksNotLiveMigratable status: 'False' type: LiveMigratable - lastProbeTime: '2022-05-17T13:59:05Z' lastTransitionTime: null status: 'True' type: AgentConnected created: true printableStatus: Running ready: true volumeSnapshotStatuses: - enabled: false name: rhel8-unknown-hornet reason: >- No VolumeSnapshotClass: Volume snapshots are not configured for this StorageClass [sno-storage] [rhel8-unknown-hornet] - enabled: false name: cloudinitdisk reason: 'Snapshot is not supported for this volumeSource type [cloudinitdisk]'
Posted https://github.com/kubevirt/kubevirt/pull/7807 to fix this.
Verified on CNV v4.11.1-3, SSH key still the same after restarting VM
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.11.1 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:8750