Bug 2087177 - Restart of VM Pod causes SSH keys to be regenerated within VM
Summary: Restart of VM Pod causes SSH keys to be regenerated within VM
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Virtualization
Version: 4.10.0
Hardware: x86_64
OS: Unspecified
high
high
Target Milestone: ---
: 4.11.1
Assignee: Antonio Cardace
QA Contact: Denys Shchedrivyi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-17 14:02 UTC by Damien Eversmann
Modified: 2022-12-01 21:10 UTC (History)
4 users (show)

Fixed In Version: virt-launcher-v4.11.0-97
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-12-01 21:10:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt kubevirt pull 7807 0 None open cloud-init: make cloud-init 'instance-id' persistent 2022-05-26 08:35:35 UTC
Red Hat Issue Tracker CNV-18365 0 None None None 2022-11-30 14:31:49 UTC

Description Damien Eversmann 2022-05-17 14:02:51 UTC
Description of problem:
If the pod hosting a VM is killed, when the pod is recreated and the VM restarted, the SSH keys have been changed thereby triggering a known_hosts mismatch error to a user logging in.


Version-Release number of selected component (if applicable):
OCP 4.10.12 (SNO); CNV 4.10.0

How reproducible:
Reliably

Steps to Reproduce:
1. Create RHEL VM from template
2. Log in to VM and accept host key... note creation dates of host keys in `/etc/ssh`
3. Stop and Start (or Restart) the VM from the Actions drop-down on the VM Details screen.
4. Log in and see known_hosts error
5. Delete known_hosts key and log in and see SSH keys with later creation dates

Actual results:
SSH keys are regenerated when VM is restarted from the UI and known_hosts error results.  This could potentially result in increased concern from the owner of a VM that crashes unexpectedly.

Expected results:
After VM reboots, user can log in the same SSH keys still exist.

Additional info:
If the VM is rebooted from "inside the VM" with `reboot` command, this problem does not exist.  Only when restarted from the OCP console.  It appears to be because restarting from the OCP console recreates the pod which changes the instance_id in cloud-init which triggers cloud-init to re-run its init processes.

Comment 2 Fabian Deutsch 2022-05-17 17:52:43 UTC
Just to summarize an offlist discussion: It seems that VMIs have an instance id. Whenever this instance ID is changing, then cloud init assumes that the VM is a new instance.
It could be that managed VMI - such as VMIs tied to VMs - should maintain it's instance ID throughout the VM life-cycle. IOWthe VM might need to provide the instance ID for the VMIs.

Comment 3 Kedar Bidarkar 2022-05-18 12:17:36 UTC
@Damien, What was the RHEL OS Version of the VM that was created from the Template?

Comment 4 Fabian Deutsch 2022-05-18 12:25:34 UTC
Please also provide the VM yaml

Comment 5 Damien Eversmann 2022-05-18 13:01:43 UTC
[cloud-user@rhel8-unknown-hornet ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.6 (Ootpa)

Comment 6 Damien Eversmann 2022-05-18 13:02:17 UTC
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
    kubemacpool.io/transaction-timestamp: '2022-05-17T13:53:55.894724909Z'
    kubevirt.io/latest-observed-api-version: v1
    kubevirt.io/storage-observed-api-version: v1alpha3
    name.os.template.kubevirt.io/rhel8.5: Red Hat Enterprise Linux 8.0 or higher
    vm.kubevirt.io/validations: |
      [
        {
          "name": "minimal-required-memory",
          "path": "jsonpath::.spec.domain.resources.requests.memory",
          "rule": "integer",
          "message": "This VM requires more memory.",
          "min": 1610612736
        }
      ]
  resourceVersion: '7477775'
  name: rhel8-unknown-hornet
  uid: a60525b4-6ec5-4389-86ef-8fc8b5244e53
  creationTimestamp: '2022-05-11T21:10:18Z'
  generation: 3
  managedFields:
    - apiVersion: kubevirt.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:name.os.template.kubevirt.io/rhel8.5': {}
            'f:vm.kubevirt.io/validations': {}
          'f:labels':
            'f:vm.kubevirt.io/template.version': {}
            'f:vm.kubevirt.io/template.namespace': {}
            'f:os.template.kubevirt.io/rhel8.5': {}
            'f:app': {}
            .: {}
            'f:vm.kubevirt.io/template.revision': {}
            'f:workload.template.kubevirt.io/server': {}
            'f:flavor.template.kubevirt.io/small': {}
            'f:vm.kubevirt.io/template': {}
        'f:spec':
          .: {}
          'f:dataVolumeTemplates': {}
          'f:template':
            .: {}
            'f:metadata':
              .: {}
              'f:annotations':
                .: {}
                'f:vm.kubevirt.io/flavor': {}
                'f:vm.kubevirt.io/os': {}
                'f:vm.kubevirt.io/workload': {}
              'f:labels':
                .: {}
                'f:flavor.template.kubevirt.io/small': {}
                'f:kubevirt.io/domain': {}
                'f:kubevirt.io/size': {}
                'f:os.template.kubevirt.io/rhel8.5': {}
                'f:vm.kubevirt.io/name': {}
                'f:workload.template.kubevirt.io/server': {}
            'f:spec':
              .: {}
              'f:accessCredentials': {}
              'f:domain':
                .: {}
                'f:cpu':
                  .: {}
                  'f:cores': {}
                  'f:sockets': {}
                  'f:threads': {}
                'f:devices':
                  .: {}
                  'f:disks': {}
                  'f:interfaces': {}
                  'f:networkInterfaceMultiqueue': {}
                  'f:rng': {}
                'f:machine':
                  .: {}
                  'f:type': {}
                'f:resources':
                  .: {}
                  'f:requests':
                    .: {}
                    'f:memory': {}
              'f:evictionStrategy': {}
              'f:hostname': {}
              'f:networks': {}
              'f:terminationGracePeriodSeconds': {}
              'f:volumes': {}
      manager: Mozilla
      operation: Update
      time: '2022-05-11T21:10:18Z'
    - apiVersion: kubevirt.io/v1alpha3
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:conditions': {}
          'f:created': {}
          'f:printableStatus': {}
          'f:ready': {}
      manager: Go-http-client
      operation: Update
      subresource: status
      time: '2022-05-17T13:54:00Z'
  namespace: damiens-vms
  labels:
    app: rhel8-unknown-hornet
    flavor.template.kubevirt.io/small: 'true'
    os.template.kubevirt.io/rhel8.5: 'true'
    vm.kubevirt.io/template: rhel8-server-small
    vm.kubevirt.io/template.namespace: openshift
    vm.kubevirt.io/template.revision: '1'
    vm.kubevirt.io/template.version: v0.19.3
    workload.template.kubevirt.io/server: 'true'
spec:
  dataVolumeTemplates:
    - apiVersion: cdi.kubevirt.io/v1beta1
      kind: DataVolume
      metadata:
        creationTimestamp: null
        name: rhel8-unknown-hornet
      spec:
        source:
          pvc:
            name: rhel8-1f05497b8847
            namespace: openshift-virtualization-os-images
        storage:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: '11362347344'
          storageClassName: sno-storage
          volumeMode: Filesystem
  running: true
  template:
    metadata:
      annotations:
        vm.kubevirt.io/flavor: small
        vm.kubevirt.io/os: rhel8
        vm.kubevirt.io/workload: server
      creationTimestamp: null
      labels:
        flavor.template.kubevirt.io/small: 'true'
        kubevirt.io/domain: rhel8-unknown-hornet
        kubevirt.io/size: small
        os.template.kubevirt.io/rhel8.5: 'true'
        vm.kubevirt.io/name: rhel8-unknown-hornet
        workload.template.kubevirt.io/server: 'true'
    spec:
      accessCredentials:
        - sshPublicKey:
            propagationMethod:
              configDrive: {}
            source:
              secret:
                secretName: authorizedsshkeys-rhel8-unknown-hornet
      domain:
        cpu:
          cores: 1
          sockets: 1
          threads: 1
        devices:
          disks:
            - bootOrder: 1
              disk:
                bus: virtio
              name: rhel8-unknown-hornet
            - disk:
                bus: virtio
              name: cloudinitdisk
          interfaces:
            - macAddress: '02:5e:0f:00:00:02'
              masquerade: {}
              name: default
          networkInterfaceMultiqueue: true
          rng: {}
        machine:
          type: pc-q35-rhel8.4.0
        resources:
          requests:
            memory: 2Gi
      evictionStrategy: LiveMigrate
      hostname: rhel8-unknown-hornet
      networks:
        - name: default
          pod: {}
      terminationGracePeriodSeconds: 180
      volumes:
        - dataVolume:
            name: rhel8-unknown-hornet
          name: rhel8-unknown-hornet
        - cloudInitConfigDrive:
            userData: |-
              #cloud-config
              user: cloud-user
              password: vgwo-gnbm-xrq3
              chpasswd: { expire: False }
          name: cloudinitdisk
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2022-05-17T13:53:58Z'
      status: 'True'
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: null
      message: >-
        cannot migrate VMI: PVC rhel8-unknown-hornet is not shared, live
        migration requires that all PVCs must be shared (using ReadWriteMany
        access mode)
      reason: DisksNotLiveMigratable
      status: 'False'
      type: LiveMigratable
    - lastProbeTime: '2022-05-17T13:59:05Z'
      lastTransitionTime: null
      status: 'True'
      type: AgentConnected
  created: true
  printableStatus: Running
  ready: true
  volumeSnapshotStatuses:
    - enabled: false
      name: rhel8-unknown-hornet
      reason: >-
        No VolumeSnapshotClass: Volume snapshots are not configured for this
        StorageClass [sno-storage] [rhel8-unknown-hornet]
    - enabled: false
      name: cloudinitdisk
      reason: 'Snapshot is not supported for this volumeSource type [cloudinitdisk]'

Comment 7 Antonio Cardace 2022-05-26 08:35:36 UTC
Posted https://github.com/kubevirt/kubevirt/pull/7807 to fix this.

Comment 8 Denys Shchedrivyi 2022-10-11 23:14:23 UTC
Verified on CNV v4.11.1-3, SSH key still the same after restarting VM

Comment 16 errata-xmlrpc 2022-12-01 21:10:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.11.1 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8750


Note You need to log in before you can comment on or make changes to this bug.