Bug 2149752

Summary: qemuAgentGetDisks cannot parse response from a guest agent running in Windows VM
Product: Red Hat Enterprise Linux 8 Reporter: Jaroslav Spanko <jspanko>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: Lili Zhu <lizhu>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.6CC: chhu, jdenemar, lmen, pkrempa, virt-maint, ymankad
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-8.0.0-12.module+el8.8.0+17545+95582d4e Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2150455 2152079 2152080 (view as bug list) Environment:
Last Closed: 2023-05-16 08:18:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2150455, 2152079, 2152080    

Description Jaroslav Spanko 2022-11-30 19:09:58 UTC
Description of problem:
RHV Host reports the following error for libvirtd 
-----
libvirtd[3267]: internal error: dependencies is missing or not an array
-----

This happens because of custom-ga-command guest-get-disks running periodically
libvirtd[3267]: Domain id=9 name='test' uuid=6801ef71-f7b0-4035-aac4-7ab5648f2dfd is tainted: custom-ga-command

Output for Windows VM is 
virsh -c qemu:///system qemu-agent-command test     '{ "execute": "guest-get-disks" }'
{"return":[{"name":"\\\\.\\PhysicalDrive0","partition":false,"address":{"serial":"4c19f070-34b4-46f7-af13-3488985e0582","bus-type":"sas","bus":0,"unit":0,"pci-controller":{"bus":0,"slot":8,"domain":0,"function":0},"dev":"\\\\.\\PhysicalDrive0","target":0}}]}

For Linux VM output if following
{"return":[{"name":"/dev/sda1","dependencies":["/dev/sda"],"partition":true},{"name":"/dev/sda3","dependencies":["/dev/sda"],"partition":true},{"name":"/dev/sda2","dependencies":["/dev/sda"],"partition":true},{"name":"/dev/sda","dependence


Version-Release number of selected component (if applicable):
libvirt-8.0.0-5.module+el8.6.0+14495+7194fa43.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Install Windows VM with newest guest agent
2. After a few minutes internal error reported periodically  


Actual results:
Reported internal error

Expected results:
No internal error reported

Additional info:
Honestly I am not sure if this should be handled by qemu-ga-agent
virjson.c
char **
virJSONValueObjectGetStringArray(virJSONValue *object, const char *key)
{
    g_auto(GStrv) ret = NULL;
    virJSONValue *data;
    size_t n;
    size_t i;

    data = virJSONValueObjectGetArray(object, key);
    if (!data) {
        virReportError(VIR_ERR_INTERNAL_ERROR,
                       _("%s is missing or not an array"),
                       key);
        return NULL;
    }

Comment 1 Jiri Denemark 2022-12-01 11:42:10 UTC
The reason is not the custom-ga-command as it would just return the reply
without trying to parse it. Libvirt calls guest-get-disks itself via a
supported API and fails to parse the response from Windows.

Comment 2 Peter Krempa 2022-12-01 12:28:45 UTC
virJSONValueObjectGetStringArray originally didn't report errors but an error was added when the function was reused further, not considering the original use.

Comment 3 Lili Zhu 2022-12-02 09:13:07 UTC
Reproduced this bug with:
libvirt-8.0.0-5.5.module+el8.6.0+16828+96e76c36.x86_64

1. prepare a win guest with guest agent device
# virsh domtime win2k19 
Time: 1669923955

2. config the libvirtd conf file
# grep -v ^# /etc/libvirt/libvirtd.conf 
log_outputs = "3:file:/var/log/libvirt/libvirtd.log"

3. restart libvirtd

4. get guest disk info
# virsh qemu-agent-command win2k19 '{ "execute": "guest-get-disks" }'
{"return":[{"name":"\\\\.\\PhysicalDrive1","partition":false,"address":{"serial":"QM00003","bus-type":"sata","bus":0,"unit":0,"pci-controller":{"bus":-1,"slot":-1,"domain":-1,"function":-1},"dev":"\\\\.\\PhysicalDrive1","target":0}},{"name":"\\\\.\\PhysicalDrive0","partition":false,"address":{"serial":"QM00001","bus-type":"sata","bus":0,"unit":0,"pci-controller":{"bus":-1,"slot":-1,"domain":-1,"function":-1},"dev":"\\\\.\\PhysicalDrive0","target":0}}]}

(There is no dependency element in win guest output)

5. check the log
# cat /var/log/libvirt/libvirtd.log 

2022-12-02 08:48:08.888+0000: 34725: error : virJSONValueObjectGetStringArray:1324 : internal error: dependencies is missing or not an array
2022-12-02 08:48:08.888+0000: 34725: error : virJSONValueObjectGetStringArray:1324 : internal error: dependencies is missing or not an array

6. prepare a rhel guest, and get the disk info
# virsh qemu-agent-command rhel8 '{ "execute": "guest-get-disks" }'
{"return":[{"name":"/dev/vda1","dependencies":["/dev/vda"],"partition":true},{"name":"/dev/vda2","dependencies":["/dev/vda"],"partition":true},{"name":"/dev/vda","dependencies":[],"partition":false,"address":{"bus-type":"virtio","bus":0,"unit":0,"pci-controller":{"bus":4,"slot":0,"domain":0,"function":0},"dev":"/dev/vda","target":0}},{"name":"/dev/dm-0","dependencies":["/dev/vda2"],"partition":false,"alias":"rhel-root"},{"name":"/dev/dm-1","dependencies":["/dev/vda2"],"partition":false,"alias":"rhel-swap"}]}

(There are dependency elements in rhel guest output)

In the function of
int qemuAgentGetDisks(qemuAgent *agent,
                      qemuAgentDiskInfo ***disks,
                      bool report_unsupported)
...
        if (virJSONValueObjectGetBoolean(entry, "partition", &disk->partition) < 0) {
            virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
                           _("'partition' missing in reply of guest-get-disks"));
            goto error;
        }

        disk->dependencies = virJSONValueObjectGetStringArray(entry, "dependencies");
        disk->alias = g_strdup(virJSONValueObjectGetString(entry, "alias"));
...

libvirt tries to parse the dependencies info no matter whether dependencies info contained in qemu agent command outputs.

Comment 4 Peter Krempa 2022-12-05 11:51:39 UTC
Fixed upstream by:

commit 3b576601dfb924bb518870a01de5d1a421cbb467
Author: Peter Krempa <pkrempa>
Date:   Thu Dec 1 17:02:42 2022 +0100

    qemuAgentGetDisks: Don't use virJSONValueObjectGetStringArray for optional data
    
    The 'dependencies' field in the return data may be missing in some
    cases. Historically 'virJSONValueObjectGetStringArray' didn't report
    error in such case, but later refactor (commit 043b50b948ef3c2 ) added
    an error in order to use it in other places too.
    
    Unfortunately this results in the error log being spammed with an
    irrelevant error in case when qemuAgentGetDisks is invoked on a VM
    running windows.
    
    Replace the use of virJSONValueObjectGetStringArray by fetching the
    array first and calling virJSONValueArrayToStringList only when we have
    an array.
    
    Fixes: 043b50b948ef3c2a4adf5fa32a93ec2589851ac6
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2149752
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Michal Privoznik <mprivozn>

commit 6765bdeaf7e9cbdb4c39d47f3b77fb28a498408a
Author: Peter Krempa <pkrempa>
Date:   Thu Dec 1 13:32:07 2022 +0100

    util: json: Split out array->strinlist conversion from virJSONValueObjectGetStringArray
    
    Introduce virJSONValueArrayToStringList which does only the conversion
    from an array to a stringlist.
    
    This will allow refactoring the callers to be more careful in case when
    they want to handle the existance of the member in the parent object
    differently.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Michal Privoznik <mprivozn>

v8.10.0-32-g3b576601df

Comment 14 Lili Zhu 2022-12-16 14:01:26 UTC
Verify this bug with:
libvirt-8.0.0-12.module+el8.8.0+17545+95582d4e.x86_64

The first 4 steps are the same with those in Comment #3

5. check the log
# cat /var/log/libvirt/libvirtd.log 
(no output)

As the testing result matches with the expected result. Mark the bug as verified.

Comment 16 errata-xmlrpc 2023-05-16 08:18:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2757