Bug 1368044

Summary: Error info should be more friendly when plug the vf to guest with <iommu model='intel'/>
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Jingjing Shao <jishao>
Component: libvirtAssignee: Andrea Bolognani <abologna>
Status: CLOSED CURRENTRELEASE QA Contact: yalzhang <yalzhang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: ---CC: dyuan, rbalakri, xuzhang, yalzhang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-12-15 07:44:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
the log of q35 none

Description Jingjing Shao 2016-08-18 08:06:29 UTC
Description of problem:
Error info should be more friendly when plug the vf to guest with <iommu model='intel'/>

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.6.0-20.el7.x86_64
libvirt-2.0.0-5.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. # lspci | grep Eth
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5764M Gigabit Ethernet PCIe (rev 10)
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

2.Prepare a guest  which is q35 machine type and with <iommu model='intel'/>
# virsh dumpxml q35
<domain type='kvm'>
  <name>q35</name>
  <uuid>0d308326-beef-4136-9386-0537185439b2</uuid>
...
  <os>
    <type arch='x86_64' machine='pc-q35-rhel7.3.0'>hvm</type>
    <boot dev='hd'/>
  </os>
...
  <iommu model='intel'/>
...

3.# virsh start q35
Domain q35 started

# virsh dumpxml q35 | grep interface -A 4
# 
# 

4.# cat vf1.xml
  <interface type='hostdev' managed='yes'>
   <mac address='52:89:bb:8c:ca:fc'/>
     <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <model type='virtio'/>  
 </interface>

# virsh attach-device q35  vf1.xml
error: Failed to attach device from vf1.xml
error: Unable to read from monitor: Connection reset by peer
Actual results:

5. Destroy the guest, Add the xml as below to the guest xml
   <interface type='hostdev' managed='yes'>
   <mac address='52:89:bb:8c:ca:fc'/>
     <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <model type='virtio'/>  
 </interface>

  # virsh edit q35
Domain q35 XML configuration edited.

# virsh start q35
error: Failed to start domain q35
error: internal error: qemu unexpectedly closed the monitor: ort=1 -device vfio-pci,host=03:10.0,id=hostdev0,bus=pci.2,addr=0x5 -device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x4 -msg timestamp=on
qemu: hardware error: Device at bus pci.2 addr 05.0 requires iommu notifier which is currently not supported by intel-iommu emulation
CPU #0:
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00010673
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000

Actual results:
According to the bug https://bugzilla.redhat.com/show_bug.cgi?id=1346920, it is as expected, But I check the http://libvirt.org/formatdomain.html#elementsIommu, 
it also has no more related info about this part. 


Expected results:
I think libvirt should give a more friendly error info or add more info in the libvirt.org

Additional info:

Comment 2 Peter Krempa 2016-08-18 08:45:51 UTC
Could you please post the VM log file (/var/log/libvirt/qemu/q35.log) after qemu crashed/failed to add the device after step4.

Comment 3 Jingjing Shao 2016-08-18 09:46:24 UTC
Created attachment 1191834 [details]
the log of q35

Comment 4 Jingjing Shao 2016-08-18 09:50:35 UTC
(In reply to Peter Krempa from comment #2)
> Could you please post the VM log file (/var/log/libvirt/qemu/q35.log) after
> qemu crashed/failed to add the device after step4.

OK, I have attached the log in comment3

Comment 5 Jaroslav Suchanek 2019-07-25 12:13:30 UTC
I would be more worried about the qemu crash, but lets see else can be done.

Comment 8 RHEL Program Management 2020-12-15 07:44:42 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 9 yalzhang@redhat.com 2020-12-16 01:48:15 UTC
Test with below packages, the result is as expected:
# rpm -q libvirt-libs qemu-kvm
libvirt-libs-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64
qemu-kvm-5.2.0-2.module+el8.4.0+9186+ec44380f.x86_64

1. start vm with "<iommu model='intel'/>";
2. hotplug hostdev interface device as below:
[root@host ~]# cat net.xml 
<interface type='hostdev' managed='yes'>
   <mac address='52:89:bb:8c:ca:fc'/>
     <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x3'/>
      </source>
 </interface>
[root@host ~]# virsh attach-device rhel net.xml 
error: Failed to attach device from net.xml
error: internal error: unable to execute QEMU command 'device_add': Device assignment is not allowed without enabling caching-mode=on for Intel IOMMU.

[root@host ~]# virsh attach-interface rhel hostdev 0000:82:10.3 --managed
error: Failed to attach interface
error: internal error: unable to execute QEMU command 'device_add': Device assignment is not allowed without enabling caching-mode=on for Intel IOMMU.

3. destroy the vm, and cold plug the interface, then try to start the vm:
[root@host ~]# virsh destroy rhel
Domain rhel destroyed

[root@host ~]# virsh attach-device rhel net.xml --config
Device attached successfully

[root@host ~]# virsh start rhel 
error: Failed to start domain rhel
error: internal error: qemu unexpectedly closed the monitor: 2020-12-16T01:46:49.200693Z qemu-kvm: We need to set caching-mode=on for intel-iommu to enable device assignment with IOMMU protection.

The bug is fixed.