Bug 1451862

Series under review upstream:
https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg02467.html

Fix included in qemu-kvm-rhev-2.9.0-9.el7

Qemu segmentation when boot guest with vhost-user using iommu .

Versions:
kernel-3.10.0-681.el7.x86_64
qemu-kvm-rhev-2.9.0-10.el7.x86_64


Scenario1: Guest act as vhost-user client
Steps:
1. Boot ovs acting as vhost-user server

2. Boot guest acting as vhost-user client , and using iommu, fail.
/usr/libexec/qemu-kvm \
-name guest=rhel7.4_nonrt \
-machine q35,kernel-irqchip=split \
-device intel-iommu,device-iotlb=on,intremap \
-cpu host \
-m 8192 \
-smp 6,sockets=1,cores=6,threads=1 \
-device pcie-root-port,id=root.1,slot=1 \
-device pcie-root-port,id=root.2,slot=2 \
-device pcie-root-port,id=root.3,slot=3 \
-object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages,share=yes,size=8589934592,host-nodes=1,policy=bind \
-numa node,nodeid=0,cpus=0-5,memdev=ram-node0 \
-drive file=/home/images_nfv-virt-rt-kvm/rhel7.4_nonrt.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=threads \
-device virtio-blk-pci,scsi=off,bus=root.1,drive=drive-virtio-disk0,id=virtio-disk0 \
-chardev socket,id=charnet1,path=/var/run/openvswitch/vhost-user0 \
-netdev vhost-user,chardev=charnet1,id=hostnet1 \
-device virtio-net-pci,mq=on,netdev=hostnet1,id=net1,mac=88:66:da:5f:dd:12,bus=root.2,iommu_platform=on,ats=on \
-chardev socket,id=charnet2,path=/var/run/openvswitch/vhost-user0 \
-netdev vhost-user,chardev=charnet2,id=hostnet2 \
-device virtio-net-pci,mq=on,netdev=hostnet2,id=net2,mac=88:66:da:5f:dd:13,bus=root.3,iommu_platform=on,ats=on \
-monitor stdio \
-vnc :2

(qemu) qemu-kvm: Failed to read msg header. Read 0 instead of 12. Original request 1040155400.
qemu-kvm: Fail to update device iotlb
qemu-kvm: Failed to set msg fds.
qemu-kvm: Fail to update device iotlb
qemu-kvm: Failed to set msg fds.
qemu-kvm: Failed to set msg fds.
qemu-kvm: Failed to set msg fds.
qemu-kvm: vhost VQ 0 ring restore failed: -1: Invalid argument (22)
qemu-kvm: Failed to set msg fds.
qemu-kvm: vhost VQ 1 ring restore failed: -1: Invalid argument (22)
qemu-kvm: Failed to read msg header. Read 0 instead of 12. Original request 1040155400.
qemu-kvm: Fail to update device iotlb
qemu-kvm: Failed to set msg fds.
qemu-kvm: Fail to update device iotlb
qemu-kvm: Failed to set msg fds.
qemu-kvm: Failed to set msg fds.
qemu-kvm: Failed to set msg fds.
qemu-kvm: vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11)
qemu-kvm: Failed to set msg fds.
qemu-kvm: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
Segmentation fault


Scenario 2: Guest act as vhost-user server
1. Boot ovs acting as vhost-user client

2. Boot guest acting as vhost-user server , and using iommu, fail.
...
-chardev socket,id=charnet1,path=/tmp/vhostuser0.sock,server \
-netdev vhost-user,chardev=charnet1,id=hostnet1 \
-device virtio-net-pci,mq=on,netdev=hostnet1,id=net1,mac=88:66:da:5f:dd:12,bus=root.2,iommu_platform=on,ats=on \
-chardev socket,id=charnet2,path=/tmp/vhostuser1.sock,server \
-netdev vhost-user,chardev=charnet2,id=hostnet2 \
-device virtio-net-pci,mq=on,netdev=hostnet2,id=net2,mac=88:66:da:5f:dd:13,bus=root.3,iommu_platform=on,ats=on \


(qemu) qemu-kvm: Failed to read msg header. Read 0 instead of 12. Original request 1948623624.
qemu-kvm: Fail to update device iotlb
qemu-kvm: Failed to set msg fds.
qemu-kvm: Fail to update device iotlb
qemu-kvm: Failed to set msg fds.
qemu-kvm: Failed to set msg fds.
qemu-kvm: Failed to set msg fds.
qemu-kvm: vhost VQ 0 ring restore failed: -1: Invalid argument (22)
qemu-kvm: Failed to set msg fds.
qemu-kvm: vhost VQ 1 ring restore failed: -1: Invalid argument (22)
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
vhost lacks feature mask 8589934592 for backend
qemu-kvm: failed to init vhost_net for queue 0
Segmentation fault



Hi Maxime, 

Could you please check the command lines? Are they correct? Thanks.


Best Regards,
Pei

Hi Pei,

The problem is that the vhost-user backend you test with does not implement IOMMU support.

That said, even if backend does not support the feature, it should not crash like this.
I think the problem should be similar with old kernel backend.
This is however not met, as QEMU & Kernel are delivered through the same channel,
so their versions are synchronized.

I'll try to provide a patch to prevent crashing when the backend does support the feature.

About testing IOMMU feature with user backend, note that it is not supported yet, neither downstream nor upstream. The support in DPDK is being implemented.

Regards,
Maxime

(In reply to Maxime Coquelin from comment #7)
> Hi Pei,
> 
> The problem is that the vhost-user backend you test with does not implement
> IOMMU support.
> 
> That said, even if backend does not support the feature, it should not crash
> like this.
> I think the problem should be similar with old kernel backend.
> This is however not met, as QEMU & Kernel are delivered through the same
> channel,
> so their versions are synchronized.
> 
> I'll try to provide a patch to prevent crashing when the backend does
> support the feature.

@pei, maybe we could file a bz with rhel7.5 flag for tracking this issue.

Best Regards,
Junyi

> 
> About testing IOMMU feature with user backend, note that it is not supported
> yet, neither downstream nor upstream. The support in DPDK is being
> implemented.
> 
> Regards,
> Maxime

(In reply to Maxime Coquelin from comment #7)
> Hi Pei,
> 
> The problem is that the vhost-user backend you test with does not implement
> IOMMU support.
> 
> That said, even if backend does not support the feature, it should not crash
> like this.
> I think the problem should be similar with old kernel backend.
> This is however not met, as QEMU & Kernel are delivered through the same
> channel,
> so their versions are synchronized.

Hi Maxime:

I rem I've tested this scenario, since vhost-kernel can fallback to userspace virtio, so it should work with a warning like "fallback to userspace". Anyway, we can let QE test in this case.

Pei, can you test qemu-kvm-rhev with IOMMU/device IOTLB enabled on 7.3 kernel?

Thanks

> 
> I'll try to provide a patch to prevent crashing when the backend does
> support the feature.
> 
> About testing IOMMU feature with user backend, note that it is not supported
> yet, neither downstream nor upstream. The support in DPDK is being
> implemented.
> 
> Regards,
> Maxime

Hi Jason,

(In reply to jason wang from comment #9)
> (In reply to Maxime Coquelin from comment #7)
> > Hi Pei,
> > 
> > The problem is that the vhost-user backend you test with does not implement
> > IOMMU support.
> > 
> > That said, even if backend does not support the feature, it should not crash
> > like this.
> > I think the problem should be similar with old kernel backend.
> > This is however not met, as QEMU & Kernel are delivered through the same
> > channel,
> > so their versions are synchronized.
> 
> Hi Maxime:
> 
> I rem I've tested this scenario, since vhost-kernel can fallback to
> userspace virtio, so it should work with a warning like "fallback to
> userspace". Anyway, we can let QE test in this case.

Thanks for the info. I'll have a look at the differences between the two backend implementations, so that we have a consistent behaviour.

Hi Jason, Maxime,

With rhel7.3, the guest can boot up in below 2 scenarios.

(1)ovs acts as vhost-user server, and vm acts as server
(2)ovs acts as vhost-user client, and vm acts as server


Versions:
3.10.0-514.25.2.el7.x86_64

Add qemu version of Comment 11.

qemu-kvm-rhev-2.9.0-10.el7.x86_64

(In reply to Maxime Coquelin from comment #10)
> Hi Jason,
> 
> (In reply to jason wang from comment #9)
> > (In reply to Maxime Coquelin from comment #7)
> > > Hi Pei,
> > > 
> > > The problem is that the vhost-user backend you test with does not implement
> > > IOMMU support.
> > > 
> > > That said, even if backend does not support the feature, it should not crash
> > > like this.
> > > I think the problem should be similar with old kernel backend.
> > > This is however not met, as QEMU & Kernel are delivered through the same
> > > channel,
> > > so their versions are synchronized.
> > 
> > Hi Maxime:
> > 
> > I rem I've tested this scenario, since vhost-kernel can fallback to
> > userspace virtio, so it should work with a warning like "fallback to
> > userspace". Anyway, we can let QE test in this case.
> 
> Thanks for the info. I'll have a look at the differences between the two
> backend implementations, so that we have a consistent behaviour.

It might not be easy since vhost-user does not have a real fallback like what vhost-kernel did (tap/macvtap could be read/wrote by userspace qemu too). Maybe we can just fail the feature negotiation in this case.

Thanks

As DPDK backend implementation is not available yet upstream,
verifying that it does not break setup without IOMMU attached is enough.

Thanks,
Maxime

Verification:

Versions:
3.10.0-685.el7.x86_64
qemu-kvm-rhev-2.9.0-12.el7.x86_64
libvirt-3.2.0-14.el7.x86_64
openvswitch-2.7.0-8.git20170530.el7fdb.x86_64
dpdk-17.05-2.el7fdb.x86_64


Testing scenario:Guest with ovs+dpdk+vhost-user (Without IOMMU)

(1)Throughput performance looks good.
With vhost-user 2 queues, the bidirectional throughput is 21.31Mpps(0 loss)

(2)Live migration with single queue works well.
20 ping-pong migration works expected, no any error.

(In reply to juzhang from comment #8)
> (In reply to Maxime Coquelin from comment #7)
> > Hi Pei,
> > 
> > The problem is that the vhost-user backend you test with does not implement
> > IOMMU support.
> > 
> > That said, even if backend does not support the feature, it should not crash
> > like this.
> > I think the problem should be similar with old kernel backend.
> > This is however not met, as QEMU & Kernel are delivered through the same
> > channel,
> > so their versions are synchronized.
> > 
> > I'll try to provide a patch to prevent crashing when the backend does
> > support the feature.
> 
> @pei, maybe we could file a bz with rhel7.5 flag for tracking this issue.

File a new bug[1] to track this issue.

[1]Bug 1464132 - Booting/shutting down VM with vhost-user and IOMMU when backend does not support IOMMU will cause qemu error

Move status of this bug to 'VERIFIED' as Comment 14, Comment 15.

QE will flow up vhost-user with vIOMMU testing when bug[1] is fixed.

[1]Bug 1335825 - [RFE] IOMMU support in DPDK for Vhost-user using IOTLB caching


Please correct me if any mistakes about status of this bug.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392