Bug 1451862

Summary: IOMMU support in QEMU for Vhost-user backend
Product: Red Hat Enterprise Linux 7 Reporter: Maxime Coquelin <maxime.coquelin>
Component: qemu-kvm-rhevAssignee: Maxime Coquelin <maxime.coquelin>
Status: CLOSED ERRATA QA Contact: Pei Zhang <pezhang>
Severity: unspecified Docs Contact:
Priority: high    
Version: 7.4CC: ailan, atragler, chayang, jasowang, jhsiao, juzhang, lmiksik, maxime.coquelin, michen, mtessun, pezhang, tli, virt-maint, xiywang
Target Milestone: rcKeywords: FutureFeature
Target Release: 7.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.9.0-9.el7 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: 1335825 Environment:
Last Closed: 2017-08-02 04:41:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 1283104, 1288337, 1335825    

Description Maxime Coquelin 2017-05-17 17:09:32 UTC
This ticket is to track the resuired changes in QEMU to support IOMMU in vhost-user backends.

QEM changes also include vhost-user spec update.

+++ This bug was initially created as a clone of Bug #1335825 +++

DPDK should properly support IOMMU when using Vhost-user, in order to allow guests to securely access vhost-user ports from user space.
This will be implemented using the IOTLB caching approach in vhost user.

Comment 2 Maxime Coquelin 2017-05-17 17:14:13 UTC
Series under review upstream:
https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg02467.html

Comment 4 Miroslav Rezanina 2017-06-08 16:27:34 UTC
Fix included in qemu-kvm-rhev-2.9.0-9.el7

Comment 6 Pei Zhang 2017-06-17 15:24:28 UTC
Qemu segmentation when boot guest with vhost-user using iommu .

Versions:
kernel-3.10.0-681.el7.x86_64
qemu-kvm-rhev-2.9.0-10.el7.x86_64


Scenario1: Guest act as vhost-user client
Steps:
1. Boot ovs acting as vhost-user server

2. Boot guest acting as vhost-user client , and using iommu, fail.
/usr/libexec/qemu-kvm \
-name guest=rhel7.4_nonrt \
-machine q35,kernel-irqchip=split \
-device intel-iommu,device-iotlb=on,intremap \
-cpu host \
-m 8192 \
-smp 6,sockets=1,cores=6,threads=1 \
-device pcie-root-port,id=root.1,slot=1 \
-device pcie-root-port,id=root.2,slot=2 \
-device pcie-root-port,id=root.3,slot=3 \
-object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages,share=yes,size=8589934592,host-nodes=1,policy=bind \
-numa node,nodeid=0,cpus=0-5,memdev=ram-node0 \
-drive file=/home/images_nfv-virt-rt-kvm/rhel7.4_nonrt.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=threads \
-device virtio-blk-pci,scsi=off,bus=root.1,drive=drive-virtio-disk0,id=virtio-disk0 \
-chardev socket,id=charnet1,path=/var/run/openvswitch/vhost-user0 \
-netdev vhost-user,chardev=charnet1,id=hostnet1 \
-device virtio-net-pci,mq=on,netdev=hostnet1,id=net1,mac=88:66:da:5f:dd:12,bus=root.2,iommu_platform=on,ats=on \
-chardev socket,id=charnet2,path=/var/run/openvswitch/vhost-user0 \
-netdev vhost-user,chardev=charnet2,id=hostnet2 \
-device virtio-net-pci,mq=on,netdev=hostnet2,id=net2,mac=88:66:da:5f:dd:13,bus=root.3,iommu_platform=on,ats=on \
-monitor stdio \
-vnc :2

(qemu) qemu-kvm: Failed to read msg header. Read 0 instead of 12. Original request 1040155400.
qemu-kvm: Fail to update device iotlb
qemu-kvm: Failed to set msg fds.
qemu-kvm: Fail to update device iotlb
qemu-kvm: Failed to set msg fds.
qemu-kvm: Failed to set msg fds.
qemu-kvm: Failed to set msg fds.
qemu-kvm: vhost VQ 0 ring restore failed: -1: Invalid argument (22)
qemu-kvm: Failed to set msg fds.
qemu-kvm: vhost VQ 1 ring restore failed: -1: Invalid argument (22)
qemu-kvm: Failed to read msg header. Read 0 instead of 12. Original request 1040155400.
qemu-kvm: Fail to update device iotlb
qemu-kvm: Failed to set msg fds.
qemu-kvm: Fail to update device iotlb
qemu-kvm: Failed to set msg fds.
qemu-kvm: Failed to set msg fds.
qemu-kvm: Failed to set msg fds.
qemu-kvm: vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11)
qemu-kvm: Failed to set msg fds.
qemu-kvm: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
Segmentation fault


Scenario 2: Guest act as vhost-user server
1. Boot ovs acting as vhost-user client

2. Boot guest acting as vhost-user server , and using iommu, fail.
...
-chardev socket,id=charnet1,path=/tmp/vhostuser0.sock,server \
-netdev vhost-user,chardev=charnet1,id=hostnet1 \
-device virtio-net-pci,mq=on,netdev=hostnet1,id=net1,mac=88:66:da:5f:dd:12,bus=root.2,iommu_platform=on,ats=on \
-chardev socket,id=charnet2,path=/tmp/vhostuser1.sock,server \
-netdev vhost-user,chardev=charnet2,id=hostnet2 \
-device virtio-net-pci,mq=on,netdev=hostnet2,id=net2,mac=88:66:da:5f:dd:13,bus=root.3,iommu_platform=on,ats=on \


(qemu) qemu-kvm: Failed to read msg header. Read 0 instead of 12. Original request 1948623624.
qemu-kvm: Fail to update device iotlb
qemu-kvm: Failed to set msg fds.
qemu-kvm: Fail to update device iotlb
qemu-kvm: Failed to set msg fds.
qemu-kvm: Failed to set msg fds.
qemu-kvm: Failed to set msg fds.
qemu-kvm: vhost VQ 0 ring restore failed: -1: Invalid argument (22)
qemu-kvm: Failed to set msg fds.
qemu-kvm: vhost VQ 1 ring restore failed: -1: Invalid argument (22)
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
Segmentation fault



Hi Maxime, 

Could you please check the command lines? Are they correct? Thanks.


Best Regards,
Pei

Comment 7 Maxime Coquelin 2017-06-19 21:19:58 UTC
Hi Pei,

The problem is that the vhost-user backend you test with does not implement IOMMU support.

That said, even if backend does not support the feature, it should not crash like this.
I think the problem should be similar with old kernel backend.
This is however not met, as QEMU & Kernel are delivered through the same channel,
so their versions are synchronized.

I'll try to provide a patch to prevent crashing when the backend does support the feature.

About testing IOMMU feature with user backend, note that it is not supported yet, neither downstream nor upstream. The support in DPDK is being implemented.

Regards,
Maxime

Comment 8 juzhang 2017-06-19 22:44:17 UTC
(In reply to Maxime Coquelin from comment #7)
> Hi Pei,
> 
> The problem is that the vhost-user backend you test with does not implement
> IOMMU support.
> 
> That said, even if backend does not support the feature, it should not crash
> like this.
> I think the problem should be similar with old kernel backend.
> This is however not met, as QEMU & Kernel are delivered through the same
> channel,
> so their versions are synchronized.
> 
> I'll try to provide a patch to prevent crashing when the backend does
> support the feature.

@pei, maybe we could file a bz with rhel7.5 flag for tracking this issue.

Best Regards,
Junyi

> 
> About testing IOMMU feature with user backend, note that it is not supported
> yet, neither downstream nor upstream. The support in DPDK is being
> implemented.
> 
> Regards,
> Maxime

Comment 9 jason wang 2017-06-20 00:17:26 UTC
(In reply to Maxime Coquelin from comment #7)
> Hi Pei,
> 
> The problem is that the vhost-user backend you test with does not implement
> IOMMU support.
> 
> That said, even if backend does not support the feature, it should not crash
> like this.
> I think the problem should be similar with old kernel backend.
> This is however not met, as QEMU & Kernel are delivered through the same
> channel,
> so their versions are synchronized.

Hi Maxime:

I rem I've tested this scenario, since vhost-kernel can fallback to userspace virtio, so it should work with a warning like "fallback to userspace". Anyway, we can let QE test in this case.

Pei, can you test qemu-kvm-rhev with IOMMU/device IOTLB enabled on 7.3 kernel?

Thanks

> 
> I'll try to provide a patch to prevent crashing when the backend does
> support the feature.
> 
> About testing IOMMU feature with user backend, note that it is not supported
> yet, neither downstream nor upstream. The support in DPDK is being
> implemented.
> 
> Regards,
> Maxime

Comment 10 Maxime Coquelin 2017-06-20 07:20:50 UTC
Hi Jason,

(In reply to jason wang from comment #9)
> (In reply to Maxime Coquelin from comment #7)
> > Hi Pei,
> > 
> > The problem is that the vhost-user backend you test with does not implement
> > IOMMU support.
> > 
> > That said, even if backend does not support the feature, it should not crash
> > like this.
> > I think the problem should be similar with old kernel backend.
> > This is however not met, as QEMU & Kernel are delivered through the same
> > channel,
> > so their versions are synchronized.
> 
> Hi Maxime:
> 
> I rem I've tested this scenario, since vhost-kernel can fallback to
> userspace virtio, so it should work with a warning like "fallback to
> userspace". Anyway, we can let QE test in this case.

Thanks for the info. I'll have a look at the differences between the two backend implementations, so that we have a consistent behaviour.

Comment 11 Pei Zhang 2017-06-20 09:03:31 UTC
Hi Jason, Maxime,

With rhel7.3, the guest can boot up in below 2 scenarios.

(1)ovs acts as vhost-user server, and vm acts as server
(2)ovs acts as vhost-user client, and vm acts as server


Versions:
3.10.0-514.25.2.el7.x86_64

Comment 12 Pei Zhang 2017-06-20 09:04:12 UTC
Add qemu version of Comment 11.

qemu-kvm-rhev-2.9.0-10.el7.x86_64

Comment 13 jason wang 2017-06-21 03:20:39 UTC
(In reply to Maxime Coquelin from comment #10)
> Hi Jason,
> 
> (In reply to jason wang from comment #9)
> > (In reply to Maxime Coquelin from comment #7)
> > > Hi Pei,
> > > 
> > > The problem is that the vhost-user backend you test with does not implement
> > > IOMMU support.
> > > 
> > > That said, even if backend does not support the feature, it should not crash
> > > like this.
> > > I think the problem should be similar with old kernel backend.
> > > This is however not met, as QEMU & Kernel are delivered through the same
> > > channel,
> > > so their versions are synchronized.
> > 
> > Hi Maxime:
> > 
> > I rem I've tested this scenario, since vhost-kernel can fallback to
> > userspace virtio, so it should work with a warning like "fallback to
> > userspace". Anyway, we can let QE test in this case.
> 
> Thanks for the info. I'll have a look at the differences between the two
> backend implementations, so that we have a consistent behaviour.

It might not be easy since vhost-user does not have a real fallback like what vhost-kernel did (tap/macvtap could be read/wrote by userspace qemu too). Maybe we can just fail the feature negotiation in this case.

Thanks

Comment 14 Maxime Coquelin 2017-06-22 10:13:26 UTC
As DPDK backend implementation is not available yet upstream,
verifying that it does not break setup without IOMMU attached is enough.

Thanks,
Maxime

Comment 15 Pei Zhang 2017-06-22 13:07:07 UTC
Verification:

Versions:
3.10.0-685.el7.x86_64
qemu-kvm-rhev-2.9.0-12.el7.x86_64
libvirt-3.2.0-14.el7.x86_64
openvswitch-2.7.0-8.git20170530.el7fdb.x86_64
dpdk-17.05-2.el7fdb.x86_64


Testing scenario:Guest with ovs+dpdk+vhost-user (Without IOMMU)

(1)Throughput performance looks good.
With vhost-user 2 queues, the bidirectional throughput is 21.31Mpps(0 loss)

(2)Live migration with single queue works well.
20 ping-pong migration works expected, no any error.

Comment 16 Pei Zhang 2017-06-22 13:40:31 UTC
(In reply to juzhang from comment #8)
> (In reply to Maxime Coquelin from comment #7)
> > Hi Pei,
> > 
> > The problem is that the vhost-user backend you test with does not implement
> > IOMMU support.
> > 
> > That said, even if backend does not support the feature, it should not crash
> > like this.
> > I think the problem should be similar with old kernel backend.
> > This is however not met, as QEMU & Kernel are delivered through the same
> > channel,
> > so their versions are synchronized.
> > 
> > I'll try to provide a patch to prevent crashing when the backend does
> > support the feature.
> 
> @pei, maybe we could file a bz with rhel7.5 flag for tracking this issue.

File a new bug[1] to track this issue.

[1]Bug 1464132 - Booting/shutting down VM with vhost-user and IOMMU when backend does not support IOMMU will cause qemu error

Comment 17 Pei Zhang 2017-06-22 13:51:32 UTC
Move status of this bug to 'VERIFIED' as Comment 14, Comment 15.

QE will flow up vhost-user with vIOMMU testing when bug[1] is fixed.

[1]Bug 1335825 - [RFE] IOMMU support in DPDK for Vhost-user using IOTLB caching


Please correct me if any mistakes about status of this bug.

Comment 19 errata-xmlrpc 2017-08-02 04:41:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392