Bug 1883857

Summary: The guest agent which is available in current RHEL 8.2 is unsupported in CNV
Product: Container Native Virtualization (CNV) Reporter: Tomas Jelinek <tjelinek>
Component: VirtualizationAssignee: sgott
Status: CLOSED ERRATA QA Contact: zhe peng <zpeng>
Severity: high Docs Contact:
Priority: urgent    
Version: 2.5.0CC: cnv-qe-bugs, ibezukh, kbidarka
Target Milestone: ---   
Target Release: 2.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: virt-operator-container-v2.5.0-79 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-17 13:24:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tomas Jelinek 2020-09-30 12:16:54 UTC 
Description of problem:
If I download the current RHEL 8.2 KVM guest image, the guest agent in it is older than the oldest supported in CNV 2.5

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. create a VM from the kvm image downloaded from here: https://access.redhat.com/downloads/content/479/ver=/rhel---8/8.2/x86_64/product-software
2. boot the VM
3. check if the GA is supported

Actual results:
GA is not supported
- lastProbeTime: '2020-09-30T09:47:51Z'
      lastTransitionTime: null
      status: 'True'
      type: AgentVersionNotSupported
  guestOSInfo: {}

Expected results:
GA is supported

Additional info:
The version of GA is 2.12.0
Comment 1 sgott 2020-09-30 12:30:01 UTC 
Kedar,

What guest agent versions are you currently testing with?
Comment 2 Kedar Bidarkar 2020-10-01 13:37:18 UTC 
@Stu, 

Answering specifically to your question,
the guest-agent currently being used in RHEL8.2 is,

[cloud-user@vm-rhel82 ~]$ rpm -qav | grep agent
qemu-guest-agent-2.12.0-99.module+el8.2.0+6870+55b789b4.2.x86_64

---

But I see something interesting, 

For the CNV-2.5.0 setup, when we check ASAP when the VMI is UP, we do see the "guestOSInfo". As seen below:

[kbidarka@kbidarka-host windows10]$ oc get csv -n openshift-cnv 
NAME                                      DISPLAY                    VERSION   REPLACES                                  PHASE
kubevirt-hyperconverged-operator.v2.5.0   OpenShift Virtualization   2.5.0     kubevirt-hyperconverged-operator.v2.4.1   Succeeded
[kbidarka@kbidarka-host windows10]$ oc get vmi vm-rhel82 -o yaml | grep "  guestOSInfo" -A 8
  guestOSInfo:
    id: rhel
    kernelRelease: 4.18.0-193.19.1.el8_2.x86_64
    kernelVersion: '#1 SMP Wed Aug 26 15:29:02 EDT 2020'
    name: Red Hat Enterprise Linux
    prettyName: Red Hat Enterprise Linux 8.2 (Ootpa)
    version: "8.2"
    versionId: "8.2"


---

But after a while, we see nothing, 'guestOSInfo: {}' , though we see the guest Agent still running inside the VMI.

[kbidarka@kbidarka-host windows10]$ oc get csv -n openshift-cnv
NAME                                      DISPLAY                    VERSION   REPLACES                                  PHASE
kubevirt-hyperconverged-operator.v2.5.0   OpenShift Virtualization   2.5.0     kubevirt-hyperconverged-operator.v2.4.1   Succeeded
[kbidarka@kbidarka-host windows10]$ oc get vmi vm-rhel82 -o yaml | grep "  guestOSInfo" -A 8
  guestOSInfo: {}


---
[kbidarka@kbidarka-host windows10]$ virtctl console vm-rhel82
Successfully connected to vm-rhel82 console. The escape sequence is ^]

[root@vm-rhel82 ~]# systemctl status qemu-guest-agent
● qemu-guest-agent.service - QEMU Guest Agent
   Loaded: loaded (/usr/lib/systemd/system/qemu-guest-agent.service; disabled; >
   Active: active (running) since Thu 2020-10-01 08:21:06 EDT; 1h 1min ago
 Main PID: 706 (qemu-ga)
    Tasks: 1 (limit: 23979)
   Memory: 2.6M
   CGroup: /system.slice/qemu-guest-agent.service
           └─706 /usr/bin/qemu-ga --method=virtio-serial --path=/dev/virtio-por>

Oct 01 08:21:06 localhost.localdomain systemd[1]: Started QEMU Guest Agent.
[root@vm-rhel82 ~]# 

---

For the 2.4.1 setup, we still do see the "guestOSInfo:" even after a while.
Comment 3 sgott 2020-10-06 21:25:58 UTC 
The disappearing guest agent info is a separate bug that's already being tracked.

As far as this issue, it was introduced in changeset 7ba5d52e26, which establishes the "GuestAgentNotSupported" condition in the first place.

We currently consider qemu GA versions in the 3.x and 4.x lines to be supported. I've taken a deep dive into the guest agent API and it appears we should be compatible with version 2.x guest agents as well.
Comment 4 sgott 2020-10-07 12:13:48 UTC 
Proposing this as a blocker due to high customer impact
Comment 5 zhe peng 2020-10-15 06:29:09 UTC 
verify with build:
virt-operator-container-v2.5.0-79

Server Version: version.Info{GitVersion:"v0.34.0-13-g2cc7d61", GitCommit:"2cc7d618eaaa1d5d5344af752dde404a62177efc", GitTreeState:"clean", BuildDate:"2020-10-10T01:15:50Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
step:

1 create a rhel82 vm
2 start vm and check qemu-guest-agent in vm
$ rpm -qa |grep agent
qemu-guest-agent-2.12.0-99.module+el8.2.0+6870+55b789b4.2.x86_64

check vmi agent info
$ oc get vmi -o yaml | grep guestOS -C 8
....
- lastProbeTime: "2020-10-15T06:23:23Z"
      lastTransitionTime: null
      status: "True"
      type: AgentConnected
    guestOSInfo:
      id: rhel
      kernelRelease: 4.18.0-193.19.1.el8_2.x86_64
      kernelVersion: '#1 SMP Wed Aug 26 15:29:02 EDT 2020'
      name: Red Hat Enterprise Linux
      prettyName: Red Hat Enterprise Linux 8.2 (Ootpa)
      version: "8.2"
      versionId: "8.2"

move to verified.
Comment 8 errata-xmlrpc 2020-11-17 13:24:55 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 2.5.0 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:5127