Bug 1970277

Summary: Attaching a virtio disk ends with an IOThreads-related error
Product: Red Hat Enterprise Linux 9 Reporter: Marcin Sobczyk <msobczyk>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED CURRENTRELEASE QA Contact: Han Han <hhan>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: bstinson, hhan, jdenemar, jsuchane, jwboyer, lmen, meili, nsoffer, pkrempa, virt-maint, xuzhang, yicui
Target Milestone: betaKeywords: Automation, Regression, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-7.5.0-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-12-07 21:57:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version: 7.5.0
Embargoed:
Bug Depends On:    
Bug Blocks: 1974096    
Attachments:
Description Flags
git bisect script none

Description Marcin Sobczyk 2021-06-10 08:26:27 UTC
Description of problem:


Version-Release number of selected component (if applicable):
libvirt-7.4.0-1.el8s.x86_64


How reproducible:
Always.

Steps to Reproduce:
Attach a disk to a VM, example disk XML:

<disk device="disk" snapshot="no" type="file">
    <source file="/rhev/data-center/mnt/192.168.202.2:_exports_nfs_share1/225cef17-e432-4b57-9607-9fb2073dda9c/images/f8c4371b-8335-4fb3-8ad4-2cea4ef35c3e/b7b81c1e-53ce-44ba-9151-af2291784593">
        <seclabel model="dac" relabel="no" type="none" />
    </source>
    <target bus="virtio" dev="vdb" />
    <serial>f8c4371b-8335-4fb3-8ad4-2cea4ef35c3e</serial>
    <driver cache="none" error_policy="stop" io="threads" iothread="1" name="qemu" type="qcow2" />
    <alias name="ua-f8c4371b-8335-4fb3-8ad4-2cea4ef35c3e" />
</disk>

Actual results:

'virDomainAttachDevice' call fails, relevant log fragment:

2021-06-10 08:16:17.361+0000: 12670: debug : virDomainAttachDevice:8325 : dom=0x7fac94005640, (VM: name=vm0, uuid=25f38807-7053-4f88-a23a-99673abac2b9), xml=<?xml version='1.0' encoding='utf-8'?>
<disk device="disk" snapshot="no" type="file">
    <source file="/rhev/data-center/mnt/192.168.202.2:_exports_nfs_share1/225cef17-e432-4b57-9607-9fb2073dda9c/images/f8c4371b-8335-4fb3-8ad4-2cea4ef35c3e/b7b81c1e-53ce-44ba-9151-af2291784593">
        <seclabel model="dac" relabel="no" type="none" />
    </source>
    <target bus="virtio" dev="vdb" />
    <serial>f8c4371b-8335-4fb3-8ad4-2cea4ef35c3e</serial>
    <driver cache="none" error_policy="stop" io="threads" iothread="1" name="qemu" type="qcow2" />
    <alias name="ua-f8c4371b-8335-4fb3-8ad4-2cea4ef35c3e" />
</disk>

2021-06-10 08:16:17.361+0000: 12670: debug : qemuDomainObjBeginJobInternal:845 : Starting job: job=modify agentJob=none asyncJob=none (vm=0x7facd80175e0 name=vm0, current job=none agentJob=none async=none)
2021-06-10 08:16:17.361+0000: 12670: debug : qemuDomainObjBeginJobInternal:892 : Started job: modify (async=none vm=0x7facd80175e0 name=vm0)
2021-06-10 08:16:17.362+0000: 12670: error : qemuValidateDomainDeviceDefDiskIOThreads:2418 : unsupported configuration: IOThreads only available for virtio pci and virtio ccw disk

Expected results:
The disk attachment succeeds.

Additional info:
Worked in libvirt 7.0.

Comment 1 Han Han 2021-06-11 08:20:12 UTC
Reproduced on v7.4.0-32-ge76ec0fe65 qemu-6.0.0-1.fc35.x86_64
when attach a disk with disk xml:
<disk device="disk" snapshot="no" type="file">
    <source file="/tmp/vdb">
    </source>
    <target bus="virtio" dev="vdb" />
    <driver io="threads" iothread="1" name="qemu" type="raw" />
</disk>

Comment 2 Han Han 2021-06-11 12:52:14 UTC
Created attachment 1790215 [details]
git bisect script

I write a script for git-bisect and then I found out the first bad commit:

3df66f27a606c5c0f27958edf7bfaaf7dcc71264 is the first bad commit
commit 3df66f27a606c5c0f27958edf7bfaaf7dcc71264
Author: Peter Krempa <pkrempa>
Date:   Fri Apr 30 16:19:15 2021 +0200

    qemu: Move iothread and s390 address validation for disk devices into the validator
    
    The "machine-loadparm-multiple-disks-nets-s390" case now requires the
    QEMU_CAPS_CCW feature to pass validation.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>
    Reviewed-by: Pavel Hrdina <phrdina>

 src/qemu/qemu_command.c  | 50 ------------------------------------------------
 src/qemu/qemu_validate.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemuxml2xmltest.c  |  2 +-
 3 files changed, 50 insertions(+), 51 deletions(-)
bisect run success

Comment 3 Sandro Bonazzola 2021-06-14 08:27:45 UTC
This is affecting oVirt 4.4.6 release based on CentOS 8 Stream.

Comment 4 Peter Krempa 2021-06-14 08:38:03 UTC
(In reply to Sandro Bonazzola from comment #3)
> This is affecting oVirt 4.4.6 release based on CentOS 8 Stream.

Note that this is filed as a rhel-9 bug.

Regardless, it will be fixed upstream in the next libvirt release. Patch is already posted for review.

Comment 5 Nir Soffer 2021-06-14 13:26:21 UTC
Peter, do we have an estimate when the fix will be available in Centos Stream?

For next oVirt release, we have a patch requiring livirt < 7.4.0:
https://gerrit.ovirt.org/c/vdsm/+/115193

because there several issues with this libvirt version. We need to know
when fixed version will be released to update our dependencies.

Comment 6 Peter Krempa 2021-06-14 13:28:55 UTC
AFAIK centos stream is/will be updated regularly to the latest release version, so I expect it to be updated few days after the next upstream release of libvirt.

Comment 7 Han Han 2021-06-16 03:34:51 UTC
Tested on libvirt v7.4.0-181-g9a51edebf8 qemu-6.0.0-5.fc35.x86_64 as comment1. PASSED

Comment 8 Han Han 2021-06-16 03:36:07 UTC
*** Bug 1972480 has been marked as a duplicate of this bug. ***

Comment 9 Peter Krempa 2021-06-16 06:45:15 UTC
Fixed upstream:

commit c29bb0fbb623af7994925bccad04060c5e27b64b
Author: Peter Krempa <pkrempa>
Date:   Thu Jun 10 16:43:36 2021 +0200

    qemu: validate: Don't check bus type in qemuValidateDomainDeviceDefDiskIOThreads
    
    IOThreads are supported with all 3 currently supported buses which can
    have virtio devices (PCI, CCW, MMIO) , so there's no need for this check.
    
    Additionally this check was buggy in the current location as on e.g.
    hotplug cases the address may not yet be assigned for the disk and thus
    a bogus error would be printed.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1970277
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

v7.4.0-161-gc29bb0fbb6

Comment 10 Han Han 2021-06-17 02:42:24 UTC
Covered by iothread_negative_test_disk_attach_delete_without_detach

Comment 14 Han Han 2021-07-26 07:05:45 UTC
rhel.cpu.iothread.negative_test.disk_attach.delete_without_detach  PASSED on libvirt-7.5.0-1.el9.x86_64