Bug 1622209

Summary: libvirt should notify the user if caching-mode isn't specified for iommu emulation in certain cases
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Bandan Das <bdas>
Component: libvirtAssignee: Virtualization Maintenance <virt-maint>
Status: CLOSED WONTFIX QA Contact: yalzhang <yalzhang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.0CC: dyuan, jtomko, lmen, peterx, pezhang, rbalakri, xuzhang, yalzhang, yanghliu
Target Milestone: rcKeywords: Triaged
Target Release: 8.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-15 07:41:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1441605    

Description Bandan Das 2018-08-24 17:52:56 UTC
Description of problem:

If caching mode isn't specified with -device intel-iommu and the user attempts to assign a host device, the guest will crash since Qemu aborts in such cases. As an example:

XML snippet:

...

  <iommu model='intel'>
      <driver intremap='on' iotlb='on'/>
    </iommu>
...

hostdev xml:
#cat pci.xml
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
      </source>
    </hostdev>

Attach the device:
#virsh attach-device rhel7.3 pci.xml

The guest will crash with the following message in the qemu log:
Device at bus pci.8 addr 00.0 requires iommu notifier which is currently not supported by intel-iommu emulation
2017-04-12 03:07:26.093+0000: shutting down, reason=crashed


It will be helpful if libvirt checks the absence of caching-mode in the XML and provides a meaningful message to the user if vfio device assignment is used.

Comment 2 Ján Tomko 2018-08-30 08:39:47 UTC
The error message specifically says:
Device at bus pci.8 addr 00.0 requires iommu notifier which is *currently* not supported by intel-iommu emulation

So it gives the impression that this can be implemented in the future.

To avoid the embarrassment of bug 1433994:
Can this configuration possibly work in the future?

Comment 3 Peter Xu 2018-08-30 09:46:19 UTC
(In reply to Ján Tomko from comment #2)
> The error message specifically says:
> Device at bus pci.8 addr 00.0 requires iommu notifier which is *currently*
> not supported by intel-iommu emulation

This is strange since at least upstream and RHEL7.6/RHEL8 should post an error like this (which is much clearer and this error should be there for a long time):

    if (!s->caching_mode && new & IOMMU_NOTIFIER_MAP) {
        error_report("We need to set caching-mode=1 for intel-iommu to enable "
                     "device assignment with IOMMU protection.");
        exit(1);
    }

I don't know where that "... currently not supported ..." come from...

> 
> So it gives the impression that this can be implemented in the future.
> 
> To avoid the embarrassment of bug 1433994:
> Can this configuration possibly work in the future?

As the (latest) error message mentioned, it will never work if without caching-mode=on.

Thanks,
Peter

Comment 4 Bandan Das 2018-09-04 20:27:31 UTC
Yes, regarding bug 1441605, I don't think comment 1 is relevant anymore. Comment 11 is and shows the actual error message.

Comment 6 Pei Zhang 2019-08-06 04:55:06 UTC
Hi Bandan,

Do we need to file a bz against this issue upon qemu component? As qemu crash when hot plug a vfio device without caching-mode=on. 

(qemu) device_add vfio-pci,host=0000:5e:00.0,bus=pci.5
We need to set caching-mode=1 for intel-iommu to enable device assignment with IOMMU protection.

In my understanding, vfio needs caching-mode=on, so this is not a valid usage, however qemu should not quit, perhaps some warning info is more friendly. What do you think? 

Thank you.

Best regards,

Pei

Comment 7 Bandan Das 2019-08-06 19:51:51 UTC
(In reply to Pei Zhang from comment #6)
> Hi Bandan,
> 
> Do we need to file a bz against this issue upon qemu component? As qemu
> crash when hot plug a vfio device without caching-mode=on. 
> 
> (qemu) device_add vfio-pci,host=0000:5e:00.0,bus=pci.5
> We need to set caching-mode=1 for intel-iommu to enable device assignment
> with IOMMU protection.
> 
> In my understanding, vfio needs caching-mode=on, so this is not a valid
> usage, however qemu should not quit, perhaps some warning info is more
> friendly. What do you think? 
> 
> Thank you.
> 
> Best regards,
> 
> Pei

We had bug 1441605 against rhel7 that was closed. I am not sure there's an easy way
to return back an error message to the monitor and the best option seems to be 
for libvirt to parse the input and return an error.

Comment 8 Pei Zhang 2019-08-07 08:04:15 UTC
(In reply to Bandan Das from comment #7)
[...]
> 
> We had bug 1441605 against rhel7 that was closed. I am not sure there's an
> easy way
> to return back an error message to the monitor and the best option seems to
> be 
> for libvirt to parse the input and return an error.

Hi Bandan,

After more testing, we found qemu shows conflict behaviors without "caching-mode=on" in different scenarios, so we filed a new bug to track it:

Bug 1738440 - For intel-iommu, qemu shows conflict behaviors between booting a guest with vfio and hot plugging vfio device 

Thank you.

Best regards,

Pei

Comment 11 RHEL Program Management 2021-02-15 07:41:58 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.