Bug 1924479 - VMI resource guest OS info is not available after a number of minutes
Summary: VMI resource guest OS info is not available after a number of minutes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Virtualization
Version: 2.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: Antonio Cardace
QA Contact: Kedar Bidarkar
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-03 07:10 UTC by Ruth Netser
Modified: 2021-07-27 14:25 UTC (History)
4 users (show)

Fixed In Version: hco-bundle-registry-container-v4.8.0-347 virt-operator-container-v4.8.0-58
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 14:23:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:2920 0 None None None 2021-07-27 14:25:37 UTC

Description Ruth Netser 2021-02-03 07:10:51 UTC
Description of problem:
When a VM is running, the VMI's resource 'Guest OS Info'  is not available after a number of minutes.

Gguest agent data provided from virtcl is available however the VMI resource one is not.

Version-Release number of selected component (if applicable):
2.6.0-106

How reproducible:
100%

Steps to Reproduce:
1. Start a VM
2. Monitor the VMI resource


Actual results:
When the VMI is running, 'Guest OS Info' is displayed.
 Guest OS Info:
    Id:              rhel
    Kernel Release:  4.18.0-240.12.1.el8_3.x86_64
    Kernel Version:  #1 SMP Wed Jan 13 06:05:15 EST 2021
    Name:            Red Hat Enterprise Linux
    Pretty Name:     Red Hat Enterprise Linux 8.3 (Ootpa)
    Version:         8.3
    Version Id:      8.3
  Interfaces:

After a number of minutes, the information is no longer available (guest agnet info is still avialable via virtctl):
   Type:                  AgentConnected
  Guest OS Info:
  Interfaces:

virtctl guestosinfo rhel8-vm
{
  "guestAgentVersion": "4.2.0",
  "hostname": "rhel8-vm",
  "os": {
    "name": "Red Hat Enterprise Linux",
    "kernelRelease": "4.18.0-240.12.1.el8_3.x86_64",
    "version": "8.3 (Ootpa)",
    "prettyName": "Red Hat Enterprise Linux 8.3 (Ootpa)",
    "versionId": "8.3",
    "kernelVersion": "#1 SMP Wed Jan 13 06:05:15 EST 2021",
    "machine": "x86_64",
    "id": "rhel"
  },
  "timezone": "EST, -18000",
  "fsInfo": {
    "disks": [
      {
        "diskName": "vda2",
        "mountPoint": "/boot/efi",
        "fileSystemType": "vfat",
        "usedBytes": 7170048,
        "totalBytes": 104634368
      },
      {
        "diskName": "vda3",
        "mountPoint": "/",
        "fileSystemType": "xfs",
        "usedBytes": 1602134016,
        "totalBytes": 32094793728
      }
    ]
  }
}

Expected results:
Guset OS info should be consistent and available both in VMI resource and virtctl.

Additional info:
 oc describe vmi rhel8-vm
Name:         rhel8-vm
Namespace:    default
Labels:       kubevirt.io/domain=rhel8-vm
              kubevirt.io/nodeName=ssp01-t6h4t-worker-0-z4qqw
              kubevirt.io/size=tiny
Annotations:  kubevirt.io/latest-observed-api-version: v1alpha3
              kubevirt.io/storage-observed-api-version: v1alpha3
API Version:  kubevirt.io/v1alpha3
Kind:         VirtualMachineInstance
Metadata:
  Creation Timestamp:  2021-02-01T16:21:33Z
  Finalizers:
    foregroundDeleteVirtualMachine
  Generation:  13
  Managed Fields:
    API Version:  kubevirt.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubevirt.io/latest-observed-api-version:
          f:kubevirt.io/storage-observed-api-version:
        f:labels:
          .:
          f:kubevirt.io/domain:
          f:kubevirt.io/nodeName:
          f:kubevirt.io/size:
        f:ownerReferences:
      f:spec:
        .:
        f:domain:
          .:
          f:cpu:
            .:
            f:cores:
            f:sockets:
            f:threads:
          f:devices:
            .:
            f:disks:
            f:interfaces:
            f:networkInterfaceMultiqueue:
            f:rng:
          f:firmware:
            .:
            f:uuid:
          f:machine:
            .:
            f:type:
          f:resources:
            .:
            f:requests:
              .:
              f:memory:
        f:evictionStrategy:
        f:networks:
        f:terminationGracePeriodSeconds:
        f:volumes:
      f:status:
        .:
        f:activePods:
          .:
          f:d1d993cb-afe4-40e6-b32f-b8d5a144f001:
        f:guestOSInfo:
        f:nodeName:
        f:qosClass:
    Manager:      virt-controller
    Operation:    Update
    Time:         2021-02-01T16:21:38Z
    API Version:  kubevirt.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:interfaces:
        f:migrationMethod:
        f:phase:
        f:volumeStatus:
    Manager:    virt-handler
    Operation:  Update
    Time:       2021-02-01T16:22:13Z
  Owner References:
    API Version:           kubevirt.io/v1alpha3
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  VirtualMachine
    Name:                  rhel8-vm
    UID:                   265981bc-7879-4f7f-9311-a310508e52f3
  Resource Version:        3720164
  Self Link:               /apis/kubevirt.io/v1alpha3/namespaces/default/virtualmachineinstances/rhel8-vm
  UID:                     8e33b9c0-c70b-4616-b0ff-42bc27f80f5f
Spec:
  Domain:
    Cpu:
      Cores:    1
      Sockets:  1
      Threads:  1
    Devices:
      Disks:
        Disk:
          Bus:  virtio
        Name:   rhel8-vm
        Disk:
          Bus:  virtio
        Name:   cloudinitdisk
      Interfaces:
        Masquerade:
        Name:                        default
      Network Interface Multiqueue:  true
      Rng:
    Features:
      Acpi:
        Enabled:  true
    Firmware:
      Uuid:  ee04d89a-59d6-5a2b-9935-6b9f7ec061fd
    Machine:
      Type:  pc-q35-rhel8.3.0
    Resources:
      Requests:
        Cpu:          100m
        Memory:       1536Mi
  Eviction Strategy:  LiveMigrate
  Networks:
    Name:  default
    Pod:
  Termination Grace Period Seconds:  180
  Volumes:
    Data Volume:
      Name:  rhel8
    Name:    rhel8-vm
    Cloud Init No Cloud:
      User Data:  #cloud-config
user: cloud-user
password: jnrt-awuf-tck8
chpasswd: { expire: False }
    Name:  cloudinitdisk
Status:
  Active Pods:
    d1d993cb-afe4-40e6-b32f-b8d5a144f001:  ssp01-t6h4t-worker-0-z4qqw
  Conditions:
    Last Probe Time:       <nil>
    Last Transition Time:  <nil>
    Status:                True
    Type:                  LiveMigratable
    Last Probe Time:       <nil>
    Last Transition Time:  2021-02-01T16:21:37Z
    Status:                True
    Type:                  Ready
    Last Probe Time:       2021-02-01T16:22:13Z
    Last Transition Time:  <nil>
    Status:                True
    Type:                  AgentConnected
  Guest OS Info:
  Interfaces:
    Interface Name:  eth0
    Ip Address:      10.131.1.124
    Ip Addresses:
      10.131.1.124
    Mac:             52:54:00:84:0a:cc
    Name:            default
  Migration Method:  BlockMigration
  Node Name:         ssp01-t6h4t-worker-0-z4qqw
  Phase:             Running
  Qos Class:         Burstable
  Volume Status:
    Name:    cloudinitdisk
    Target:  vdb
    Name:    rhel8-vm
    Target:  vda
Events:
  Type    Reason            Age                   From                         Message
  ----    ------            ----                  ----                         -------
  Normal  SuccessfulCreate  2m18s                 disruptionbudget-controller  Created PodDisruptionBudget kubevirt-disruption-budget-qhxv5
  Normal  SuccessfulCreate  2m18s                 virtualmachine-controller    Created virtual machine pod virt-launcher-rhel8-vm-x8rh9
  Normal  Started           2m13s                 virt-handler                 VirtualMachineInstance started.
  Normal  Created           35s (x14 over 2m13s)  virt-handler                 VirtualMachineInstance defined.

Comment 1 Kedar Bidarkar 2021-02-24 13:47:30 UTC
Retest this quickly with CNV-v4.8.0, once guest-agent stuff is also fixed.

Comment 3 Kedar Bidarkar 2021-04-05 12:18:33 UTC
This bug is still seen with the very latest CNV-v4.8.0 builds.

Comment 4 Antonio Cardace 2021-04-27 08:38:37 UTC
PR in review which fixes this https://github.com/kubevirt/kubevirt/pull/5539.

Comment 5 Antonio Cardace 2021-04-27 11:28:58 UTC
Fixed by https://github.com/kubevirt/kubevirt/pull/5428 which was merged 15 mins before I posted my PR.

Comment 7 Kedar Bidarkar 2021-05-31 15:11:56 UTC
Summary: LGTM, 

a) Created 2 VM's, one running since 43+hrs and the other since 24+hrs
b)As seen below even after 43hrs or 24hrs, we can cotninue to get the "Guest OS Info" from "oc describe vmi <vm-name>"



[kbidarka@localhost secureboot]$ oc describe vmi vm2-rhel84 -n default | grep -A 7 "Guest OS Info"
  Guest OS Info:
    Id:              rhel
    Kernel Release:  4.18.0-287.el8.dt4.x86_64
    Kernel Version:  #1 SMP Thu Feb 18 13:31:55 EST 2021
    Name:            Red Hat Enterprise Linux
    Pretty Name:     Red Hat Enterprise Linux 8.4 Beta (Ootpa)
    Version:         8.4
    Version Id:      8.4
[kbidarka@localhost secureboot]$ oc get vmi vm2-rhel84 -n default
NAME         AGE   PHASE     IP            NODENAME
vm2-rhel84   43h   Running   10.xxx.y.zz   node-05.redhat.com


[kbidarka@localhost secureboot]$ oc describe vmi vm2-rhel84 -n oren-test | grep -A 7 "Guest OS Info"
  Guest OS Info:
    Id:              rhel
    Kernel Release:  4.18.0-287.el8.dt4.x86_64
    Kernel Version:  #1 SMP Thu Feb 18 13:31:55 EST 2021
    Name:            Red Hat Enterprise Linux
    Pretty Name:     Red Hat Enterprise Linux 8.4 Beta (Ootpa)
    Version:         8.4
    Version Id:      8.4
[kbidarka@localhost secureboot]$ oc get vmi vm2-rhel84 -n oren-test
NAME         AGE   PHASE     IP            NODENAME
vm2-rhel84   24h   Running   10.xxx.y.mm   node-06.redhat.com


In the past we have issues after Live migration.
Plan is to live-migrate these 2 vm's wait for 24hrs and then move this bug to VERIFIED State.

This is just to double-check, live migration scenario as well.

Will update the Live migration updates shortly.

Comment 8 Kedar Bidarkar 2021-05-31 15:26:44 UTC
Summary: "we can fetch Guest OsInfo" Output even after Live Migration.

[kbidarka@localhost migration]$ oc get vmi --all-namespaces
NAMESPACE   NAME         AGE   PHASE     IP            NODENAME
default     vm2-rhel84   43h   Running   10.xxx.y.zz   node-05.redhat.com
oren-test   vm2-rhel84   24h   Running   10.xxx.y.mm   node-06.redhat.com


[kbidarka@localhost migration]$ cat migration-job2.yaml
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachineInstanceMigration
metadata:
  creationTimestamp: null
  name: job22
  namespace: default
spec:
  vmiName: vm2-rhel84
status: {}

[kbidarka@localhost migration]$ cat migration-job2-oren.yaml
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachineInstanceMigration
metadata:
  creationTimestamp: null
  name: job33
  namespace: oren-test
spec:
  vmiName: vm2-rhel84
status: {}


[kbidarka@localhost migration]$ oc apply -f migration-job2.yaml
virtualmachineinstancemigration.kubevirt.io/job22 created
[kbidarka@localhost migration]$ oc apply -f migration-job2-oren.yaml
virtualmachineinstancemigration.kubevirt.io/job33 created

[kbidarka@localhost migration]$ oc get vmi --all-namespaces
NAMESPACE   NAME         AGE   PHASE     IP             NODENAME
default     vm2-rhel84   44h   Running   10.xxx.y.cc   node-07.redhat.com
oren-test   vm2-rhel84   24h   Running   10.xxx.y.dd   node-05.redhat.com

[kbidarka@localhost migration]$ oc describe vmi vm2-rhel84 -n default | grep -A 7 "Guest OS Info"
  Guest OS Info:
    Id:              rhel
    Kernel Release:  4.18.0-287.el8.dt4.x86_64
    Kernel Version:  #1 SMP Thu Feb 18 13:31:55 EST 2021
    Name:            Red Hat Enterprise Linux
    Pretty Name:     Red Hat Enterprise Linux 8.4 Beta (Ootpa)
    Version:         8.4
    Version Id:      8.4
    
[kbidarka@localhost migration]$ oc describe vmi vm2-rhel84 -n oren-test | grep -A 7 "Guest OS Info"
  Guest OS Info:
    Id:              rhel
    Kernel Release:  4.18.0-287.el8.dt4.x86_64
    Kernel Version:  #1 SMP Thu Feb 18 13:31:55 EST 2021
    Name:            Red Hat Enterprise Linux
    Pretty Name:     Red Hat Enterprise Linux 8.4 Beta (Ootpa)
    Version:         8.4
    Version Id:      8.4


Will update here after 24hrs

Comment 9 Kedar Bidarkar 2021-06-01 10:14:27 UTC
[kbidarka@localhost secureboot]$ oc get vmi -n default 
NAME                AGE     PHASE     IP             NODENAME
vm2-rhel84          2d14h   Running   kk.xxx.y.zzz   node-07.redhat.com
vm2-rhel84-secref   17h     Running   kk.xxx.y.zyz   node-07.redhat.com



[kbidarka@localhost secureboot]$ oc describe vmi vm2-rhel84 -n default | grep -A 10 "Guest OS Info"
  Guest OS Info:
    Id:              rhel
    Kernel Release:  4.18.0-287.el8.dt4.x86_64
    Kernel Version:  #1 SMP Thu Feb 18 13:31:55 EST 2021
    Name:            Red Hat Enterprise Linux
    Pretty Name:     Red Hat Enterprise Linux 8.4 Beta (Ootpa)
    Version:         8.4
    Version Id:      8.4

---

[kbidarka@localhost secureboot]$ oc get vmi -n oren-test
NAME         AGE   PHASE     IP            NODENAME
vm2-rhel84   43h   Running   kk.xxx.y.mmm   node-05.redhat.com


[kbidarka@localhost secureboot]$ oc describe vmi vm2-rhel84 -n oren-test | grep -A 10 "Guest OS Info"
  Guest OS Info:
    Id:              rhel
    Kernel Release:  4.18.0-287.el8.dt4.x86_64
    Kernel Version:  #1 SMP Thu Feb 18 13:31:55 EST 2021
    Name:            Red Hat Enterprise Linux
    Pretty Name:     Red Hat Enterprise Linux 8.4 Beta (Ootpa)
    Version:         8.4
    Version Id:      8.4


Summary: "we can continue to fetch Guest OsInfo" Output even after Live Migration and also more than 20+hrs.
Looking at above output info.

Comment 12 errata-xmlrpc 2021-07-27 14:23:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.8.0 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2920


Note You need to log in before you can comment on or make changes to this bug.