Description of problem: When a VM is running, the VMI's resource 'Guest OS Info' is not available after a number of minutes. Gguest agent data provided from virtcl is available however the VMI resource one is not. Version-Release number of selected component (if applicable): 2.6.0-106 How reproducible: 100% Steps to Reproduce: 1. Start a VM 2. Monitor the VMI resource Actual results: When the VMI is running, 'Guest OS Info' is displayed. Guest OS Info: Id: rhel Kernel Release: 4.18.0-240.12.1.el8_3.x86_64 Kernel Version: #1 SMP Wed Jan 13 06:05:15 EST 2021 Name: Red Hat Enterprise Linux Pretty Name: Red Hat Enterprise Linux 8.3 (Ootpa) Version: 8.3 Version Id: 8.3 Interfaces: After a number of minutes, the information is no longer available (guest agnet info is still avialable via virtctl): Type: AgentConnected Guest OS Info: Interfaces: virtctl guestosinfo rhel8-vm { "guestAgentVersion": "4.2.0", "hostname": "rhel8-vm", "os": { "name": "Red Hat Enterprise Linux", "kernelRelease": "4.18.0-240.12.1.el8_3.x86_64", "version": "8.3 (Ootpa)", "prettyName": "Red Hat Enterprise Linux 8.3 (Ootpa)", "versionId": "8.3", "kernelVersion": "#1 SMP Wed Jan 13 06:05:15 EST 2021", "machine": "x86_64", "id": "rhel" }, "timezone": "EST, -18000", "fsInfo": { "disks": [ { "diskName": "vda2", "mountPoint": "/boot/efi", "fileSystemType": "vfat", "usedBytes": 7170048, "totalBytes": 104634368 }, { "diskName": "vda3", "mountPoint": "/", "fileSystemType": "xfs", "usedBytes": 1602134016, "totalBytes": 32094793728 } ] } } Expected results: Guset OS info should be consistent and available both in VMI resource and virtctl. Additional info: oc describe vmi rhel8-vm Name: rhel8-vm Namespace: default Labels: kubevirt.io/domain=rhel8-vm kubevirt.io/nodeName=ssp01-t6h4t-worker-0-z4qqw kubevirt.io/size=tiny Annotations: kubevirt.io/latest-observed-api-version: v1alpha3 kubevirt.io/storage-observed-api-version: v1alpha3 API Version: kubevirt.io/v1alpha3 Kind: VirtualMachineInstance Metadata: Creation Timestamp: 2021-02-01T16:21:33Z Finalizers: foregroundDeleteVirtualMachine Generation: 13 Managed Fields: API Version: kubevirt.io/v1alpha3 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubevirt.io/latest-observed-api-version: f:kubevirt.io/storage-observed-api-version: f:labels: .: f:kubevirt.io/domain: f:kubevirt.io/nodeName: f:kubevirt.io/size: f:ownerReferences: f:spec: .: f:domain: .: f:cpu: .: f:cores: f:sockets: f:threads: f:devices: .: f:disks: f:interfaces: f:networkInterfaceMultiqueue: f:rng: f:firmware: .: f:uuid: f:machine: .: f:type: f:resources: .: f:requests: .: f:memory: f:evictionStrategy: f:networks: f:terminationGracePeriodSeconds: f:volumes: f:status: .: f:activePods: .: f:d1d993cb-afe4-40e6-b32f-b8d5a144f001: f:guestOSInfo: f:nodeName: f:qosClass: Manager: virt-controller Operation: Update Time: 2021-02-01T16:21:38Z API Version: kubevirt.io/v1alpha3 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: f:interfaces: f:migrationMethod: f:phase: f:volumeStatus: Manager: virt-handler Operation: Update Time: 2021-02-01T16:22:13Z Owner References: API Version: kubevirt.io/v1alpha3 Block Owner Deletion: true Controller: true Kind: VirtualMachine Name: rhel8-vm UID: 265981bc-7879-4f7f-9311-a310508e52f3 Resource Version: 3720164 Self Link: /apis/kubevirt.io/v1alpha3/namespaces/default/virtualmachineinstances/rhel8-vm UID: 8e33b9c0-c70b-4616-b0ff-42bc27f80f5f Spec: Domain: Cpu: Cores: 1 Sockets: 1 Threads: 1 Devices: Disks: Disk: Bus: virtio Name: rhel8-vm Disk: Bus: virtio Name: cloudinitdisk Interfaces: Masquerade: Name: default Network Interface Multiqueue: true Rng: Features: Acpi: Enabled: true Firmware: Uuid: ee04d89a-59d6-5a2b-9935-6b9f7ec061fd Machine: Type: pc-q35-rhel8.3.0 Resources: Requests: Cpu: 100m Memory: 1536Mi Eviction Strategy: LiveMigrate Networks: Name: default Pod: Termination Grace Period Seconds: 180 Volumes: Data Volume: Name: rhel8 Name: rhel8-vm Cloud Init No Cloud: User Data: #cloud-config user: cloud-user password: jnrt-awuf-tck8 chpasswd: { expire: False } Name: cloudinitdisk Status: Active Pods: d1d993cb-afe4-40e6-b32f-b8d5a144f001: ssp01-t6h4t-worker-0-z4qqw Conditions: Last Probe Time: <nil> Last Transition Time: <nil> Status: True Type: LiveMigratable Last Probe Time: <nil> Last Transition Time: 2021-02-01T16:21:37Z Status: True Type: Ready Last Probe Time: 2021-02-01T16:22:13Z Last Transition Time: <nil> Status: True Type: AgentConnected Guest OS Info: Interfaces: Interface Name: eth0 Ip Address: 10.131.1.124 Ip Addresses: 10.131.1.124 Mac: 52:54:00:84:0a:cc Name: default Migration Method: BlockMigration Node Name: ssp01-t6h4t-worker-0-z4qqw Phase: Running Qos Class: Burstable Volume Status: Name: cloudinitdisk Target: vdb Name: rhel8-vm Target: vda Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 2m18s disruptionbudget-controller Created PodDisruptionBudget kubevirt-disruption-budget-qhxv5 Normal SuccessfulCreate 2m18s virtualmachine-controller Created virtual machine pod virt-launcher-rhel8-vm-x8rh9 Normal Started 2m13s virt-handler VirtualMachineInstance started. Normal Created 35s (x14 over 2m13s) virt-handler VirtualMachineInstance defined.
Retest this quickly with CNV-v4.8.0, once guest-agent stuff is also fixed.
This bug is still seen with the very latest CNV-v4.8.0 builds.
PR in review which fixes this https://github.com/kubevirt/kubevirt/pull/5539.
Fixed by https://github.com/kubevirt/kubevirt/pull/5428 which was merged 15 mins before I posted my PR.
https://github.com/kubevirt/kubevirt/pull/5428 was reverted, https://github.com/kubevirt/kubevirt/pull/5560 merged instead
Summary: LGTM, a) Created 2 VM's, one running since 43+hrs and the other since 24+hrs b)As seen below even after 43hrs or 24hrs, we can cotninue to get the "Guest OS Info" from "oc describe vmi <vm-name>" [kbidarka@localhost secureboot]$ oc describe vmi vm2-rhel84 -n default | grep -A 7 "Guest OS Info" Guest OS Info: Id: rhel Kernel Release: 4.18.0-287.el8.dt4.x86_64 Kernel Version: #1 SMP Thu Feb 18 13:31:55 EST 2021 Name: Red Hat Enterprise Linux Pretty Name: Red Hat Enterprise Linux 8.4 Beta (Ootpa) Version: 8.4 Version Id: 8.4 [kbidarka@localhost secureboot]$ oc get vmi vm2-rhel84 -n default NAME AGE PHASE IP NODENAME vm2-rhel84 43h Running 10.xxx.y.zz node-05.redhat.com [kbidarka@localhost secureboot]$ oc describe vmi vm2-rhel84 -n oren-test | grep -A 7 "Guest OS Info" Guest OS Info: Id: rhel Kernel Release: 4.18.0-287.el8.dt4.x86_64 Kernel Version: #1 SMP Thu Feb 18 13:31:55 EST 2021 Name: Red Hat Enterprise Linux Pretty Name: Red Hat Enterprise Linux 8.4 Beta (Ootpa) Version: 8.4 Version Id: 8.4 [kbidarka@localhost secureboot]$ oc get vmi vm2-rhel84 -n oren-test NAME AGE PHASE IP NODENAME vm2-rhel84 24h Running 10.xxx.y.mm node-06.redhat.com In the past we have issues after Live migration. Plan is to live-migrate these 2 vm's wait for 24hrs and then move this bug to VERIFIED State. This is just to double-check, live migration scenario as well. Will update the Live migration updates shortly.
Summary: "we can fetch Guest OsInfo" Output even after Live Migration. [kbidarka@localhost migration]$ oc get vmi --all-namespaces NAMESPACE NAME AGE PHASE IP NODENAME default vm2-rhel84 43h Running 10.xxx.y.zz node-05.redhat.com oren-test vm2-rhel84 24h Running 10.xxx.y.mm node-06.redhat.com [kbidarka@localhost migration]$ cat migration-job2.yaml apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachineInstanceMigration metadata: creationTimestamp: null name: job22 namespace: default spec: vmiName: vm2-rhel84 status: {} [kbidarka@localhost migration]$ cat migration-job2-oren.yaml apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachineInstanceMigration metadata: creationTimestamp: null name: job33 namespace: oren-test spec: vmiName: vm2-rhel84 status: {} [kbidarka@localhost migration]$ oc apply -f migration-job2.yaml virtualmachineinstancemigration.kubevirt.io/job22 created [kbidarka@localhost migration]$ oc apply -f migration-job2-oren.yaml virtualmachineinstancemigration.kubevirt.io/job33 created [kbidarka@localhost migration]$ oc get vmi --all-namespaces NAMESPACE NAME AGE PHASE IP NODENAME default vm2-rhel84 44h Running 10.xxx.y.cc node-07.redhat.com oren-test vm2-rhel84 24h Running 10.xxx.y.dd node-05.redhat.com [kbidarka@localhost migration]$ oc describe vmi vm2-rhel84 -n default | grep -A 7 "Guest OS Info" Guest OS Info: Id: rhel Kernel Release: 4.18.0-287.el8.dt4.x86_64 Kernel Version: #1 SMP Thu Feb 18 13:31:55 EST 2021 Name: Red Hat Enterprise Linux Pretty Name: Red Hat Enterprise Linux 8.4 Beta (Ootpa) Version: 8.4 Version Id: 8.4 [kbidarka@localhost migration]$ oc describe vmi vm2-rhel84 -n oren-test | grep -A 7 "Guest OS Info" Guest OS Info: Id: rhel Kernel Release: 4.18.0-287.el8.dt4.x86_64 Kernel Version: #1 SMP Thu Feb 18 13:31:55 EST 2021 Name: Red Hat Enterprise Linux Pretty Name: Red Hat Enterprise Linux 8.4 Beta (Ootpa) Version: 8.4 Version Id: 8.4 Will update here after 24hrs
[kbidarka@localhost secureboot]$ oc get vmi -n default NAME AGE PHASE IP NODENAME vm2-rhel84 2d14h Running kk.xxx.y.zzz node-07.redhat.com vm2-rhel84-secref 17h Running kk.xxx.y.zyz node-07.redhat.com [kbidarka@localhost secureboot]$ oc describe vmi vm2-rhel84 -n default | grep -A 10 "Guest OS Info" Guest OS Info: Id: rhel Kernel Release: 4.18.0-287.el8.dt4.x86_64 Kernel Version: #1 SMP Thu Feb 18 13:31:55 EST 2021 Name: Red Hat Enterprise Linux Pretty Name: Red Hat Enterprise Linux 8.4 Beta (Ootpa) Version: 8.4 Version Id: 8.4 --- [kbidarka@localhost secureboot]$ oc get vmi -n oren-test NAME AGE PHASE IP NODENAME vm2-rhel84 43h Running kk.xxx.y.mmm node-05.redhat.com [kbidarka@localhost secureboot]$ oc describe vmi vm2-rhel84 -n oren-test | grep -A 10 "Guest OS Info" Guest OS Info: Id: rhel Kernel Release: 4.18.0-287.el8.dt4.x86_64 Kernel Version: #1 SMP Thu Feb 18 13:31:55 EST 2021 Name: Red Hat Enterprise Linux Pretty Name: Red Hat Enterprise Linux 8.4 Beta (Ootpa) Version: 8.4 Version Id: 8.4 Summary: "we can continue to fetch Guest OsInfo" Output even after Live Migration and also more than 20+hrs. Looking at above output info.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.8.0 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2920