RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2149752 - qemuAgentGetDisks cannot parse response from a guest agent running in Windows VM
Summary: qemuAgentGetDisks cannot parse response from a guest agent running in Windows VM
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: libvirt
Version: 8.6
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: Lili Zhu
URL:
Whiteboard:
Depends On:
Blocks: 2150455 2152079 2152080
TreeView+ depends on / blocked
 
Reported: 2022-11-30 19:09 UTC by Jaroslav Spanko
Modified: 2023-05-16 09:00 UTC (History)
6 users (show)

Fixed In Version: libvirt-8.0.0-12.module+el8.8.0+17545+95582d4e
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2150455 2152079 2152080 (view as bug list)
Environment:
Last Closed: 2023-05-16 08:18:36 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-140974 0 None None None 2022-11-30 19:19:15 UTC
Red Hat Knowledge Base (Solution) 6988712 0 None None None 2022-12-01 17:43:33 UTC
Red Hat Product Errata RHSA-2023:2757 0 None None None 2023-05-16 08:19:53 UTC

Description Jaroslav Spanko 2022-11-30 19:09:58 UTC
Description of problem:
RHV Host reports the following error for libvirtd 
-----
libvirtd[3267]: internal error: dependencies is missing or not an array
-----

This happens because of custom-ga-command guest-get-disks running periodically
libvirtd[3267]: Domain id=9 name='test' uuid=6801ef71-f7b0-4035-aac4-7ab5648f2dfd is tainted: custom-ga-command

Output for Windows VM is 
virsh -c qemu:///system qemu-agent-command test     '{ "execute": "guest-get-disks" }'
{"return":[{"name":"\\\\.\\PhysicalDrive0","partition":false,"address":{"serial":"4c19f070-34b4-46f7-af13-3488985e0582","bus-type":"sas","bus":0,"unit":0,"pci-controller":{"bus":0,"slot":8,"domain":0,"function":0},"dev":"\\\\.\\PhysicalDrive0","target":0}}]}

For Linux VM output if following
{"return":[{"name":"/dev/sda1","dependencies":["/dev/sda"],"partition":true},{"name":"/dev/sda3","dependencies":["/dev/sda"],"partition":true},{"name":"/dev/sda2","dependencies":["/dev/sda"],"partition":true},{"name":"/dev/sda","dependence


Version-Release number of selected component (if applicable):
libvirt-8.0.0-5.module+el8.6.0+14495+7194fa43.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Install Windows VM with newest guest agent
2. After a few minutes internal error reported periodically  


Actual results:
Reported internal error

Expected results:
No internal error reported

Additional info:
Honestly I am not sure if this should be handled by qemu-ga-agent
virjson.c
char **
virJSONValueObjectGetStringArray(virJSONValue *object, const char *key)
{
    g_auto(GStrv) ret = NULL;
    virJSONValue *data;
    size_t n;
    size_t i;

    data = virJSONValueObjectGetArray(object, key);
    if (!data) {
        virReportError(VIR_ERR_INTERNAL_ERROR,
                       _("%s is missing or not an array"),
                       key);
        return NULL;
    }

Comment 1 Jiri Denemark 2022-12-01 11:42:10 UTC
The reason is not the custom-ga-command as it would just return the reply
without trying to parse it. Libvirt calls guest-get-disks itself via a
supported API and fails to parse the response from Windows.

Comment 2 Peter Krempa 2022-12-01 12:28:45 UTC
virJSONValueObjectGetStringArray originally didn't report errors but an error was added when the function was reused further, not considering the original use.

Comment 3 Lili Zhu 2022-12-02 09:13:07 UTC
Reproduced this bug with:
libvirt-8.0.0-5.5.module+el8.6.0+16828+96e76c36.x86_64

1. prepare a win guest with guest agent device
# virsh domtime win2k19 
Time: 1669923955

2. config the libvirtd conf file
# grep -v ^# /etc/libvirt/libvirtd.conf 
log_outputs = "3:file:/var/log/libvirt/libvirtd.log"

3. restart libvirtd

4. get guest disk info
# virsh qemu-agent-command win2k19 '{ "execute": "guest-get-disks" }'
{"return":[{"name":"\\\\.\\PhysicalDrive1","partition":false,"address":{"serial":"QM00003","bus-type":"sata","bus":0,"unit":0,"pci-controller":{"bus":-1,"slot":-1,"domain":-1,"function":-1},"dev":"\\\\.\\PhysicalDrive1","target":0}},{"name":"\\\\.\\PhysicalDrive0","partition":false,"address":{"serial":"QM00001","bus-type":"sata","bus":0,"unit":0,"pci-controller":{"bus":-1,"slot":-1,"domain":-1,"function":-1},"dev":"\\\\.\\PhysicalDrive0","target":0}}]}

(There is no dependency element in win guest output)

5. check the log
# cat /var/log/libvirt/libvirtd.log 

2022-12-02 08:48:08.888+0000: 34725: error : virJSONValueObjectGetStringArray:1324 : internal error: dependencies is missing or not an array
2022-12-02 08:48:08.888+0000: 34725: error : virJSONValueObjectGetStringArray:1324 : internal error: dependencies is missing or not an array

6. prepare a rhel guest, and get the disk info
# virsh qemu-agent-command rhel8 '{ "execute": "guest-get-disks" }'
{"return":[{"name":"/dev/vda1","dependencies":["/dev/vda"],"partition":true},{"name":"/dev/vda2","dependencies":["/dev/vda"],"partition":true},{"name":"/dev/vda","dependencies":[],"partition":false,"address":{"bus-type":"virtio","bus":0,"unit":0,"pci-controller":{"bus":4,"slot":0,"domain":0,"function":0},"dev":"/dev/vda","target":0}},{"name":"/dev/dm-0","dependencies":["/dev/vda2"],"partition":false,"alias":"rhel-root"},{"name":"/dev/dm-1","dependencies":["/dev/vda2"],"partition":false,"alias":"rhel-swap"}]}

(There are dependency elements in rhel guest output)

In the function of
int qemuAgentGetDisks(qemuAgent *agent,
                      qemuAgentDiskInfo ***disks,
                      bool report_unsupported)
...
        if (virJSONValueObjectGetBoolean(entry, "partition", &disk->partition) < 0) {
            virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
                           _("'partition' missing in reply of guest-get-disks"));
            goto error;
        }

        disk->dependencies = virJSONValueObjectGetStringArray(entry, "dependencies");
        disk->alias = g_strdup(virJSONValueObjectGetString(entry, "alias"));
...

libvirt tries to parse the dependencies info no matter whether dependencies info contained in qemu agent command outputs.

Comment 4 Peter Krempa 2022-12-05 11:51:39 UTC
Fixed upstream by:

commit 3b576601dfb924bb518870a01de5d1a421cbb467
Author: Peter Krempa <pkrempa>
Date:   Thu Dec 1 17:02:42 2022 +0100

    qemuAgentGetDisks: Don't use virJSONValueObjectGetStringArray for optional data
    
    The 'dependencies' field in the return data may be missing in some
    cases. Historically 'virJSONValueObjectGetStringArray' didn't report
    error in such case, but later refactor (commit 043b50b948ef3c2 ) added
    an error in order to use it in other places too.
    
    Unfortunately this results in the error log being spammed with an
    irrelevant error in case when qemuAgentGetDisks is invoked on a VM
    running windows.
    
    Replace the use of virJSONValueObjectGetStringArray by fetching the
    array first and calling virJSONValueArrayToStringList only when we have
    an array.
    
    Fixes: 043b50b948ef3c2a4adf5fa32a93ec2589851ac6
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2149752
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Michal Privoznik <mprivozn>

commit 6765bdeaf7e9cbdb4c39d47f3b77fb28a498408a
Author: Peter Krempa <pkrempa>
Date:   Thu Dec 1 13:32:07 2022 +0100

    util: json: Split out array->strinlist conversion from virJSONValueObjectGetStringArray
    
    Introduce virJSONValueArrayToStringList which does only the conversion
    from an array to a stringlist.
    
    This will allow refactoring the callers to be more careful in case when
    they want to handle the existance of the member in the parent object
    differently.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Michal Privoznik <mprivozn>

v8.10.0-32-g3b576601df

Comment 14 Lili Zhu 2022-12-16 14:01:26 UTC
Verify this bug with:
libvirt-8.0.0-12.module+el8.8.0+17545+95582d4e.x86_64

The first 4 steps are the same with those in Comment #3

5. check the log
# cat /var/log/libvirt/libvirtd.log 
(no output)

As the testing result matches with the expected result. Mark the bug as verified.

Comment 16 errata-xmlrpc 2023-05-16 08:18:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2757


Note You need to log in before you can comment on or make changes to this bug.