Bug 1499127

Hi Pei,

I will try to reproduce the issue.
Does it works well when having iommu=pt and intel_iommu=on in kernel cmdline?

Thanks,
Maxime

Other question is does the same setup works with vhost-kernel backend,
or Qemu & backend also hang?

I reproduce an issue with user backend, but it does not hang neither the guest nor QEMU.

The problem is that QEMU sends IOTLB updates not covering the addresses passed with the IOTLB miss requests.
So the vring addresses are never translated on backend side and the rings are not started.

For example, the backend needs to translate iova 0x179d7b240, corresponding to the desc ring start address. On Qemu side, we can see:
vhost_device_iotlb_miss: miss iova 179d7b240
vhost_device_iotlb_miss: iotlb update iova 240 - len 9fdc0

(In reply to Maxime Coquelin from comment #2)
> Hi Pei,
> 
> I will try to reproduce the issue.
> Does it works well when having iommu=pt and intel_iommu=on in kernel cmdline?
> 
Hi Maxime,

With iommu=pt and intel_iommu=on in kernel cmdline, still can hit hang issue.


(In reply to Maxime Coquelin from comment #3)
> Other question is does the same setup works with vhost-kernel backend,
> or Qemu & backend also hang?

1. With same steps but vhost-user network without "iommu_platform=on,ats=on"[1], everything works well, both qemu & testpmd backend works well.

[1]
-chardev socket,id=char0,path=/tmp/vhost-user1 \
-device virtio-net-pci,netdev=mynet1,mac=aa:66:da:5f:dd:11,bus=root.1 \
-netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \

Related full command:
# /usr/libexec/qemu-kvm -name rhel7.5 -M q35,kernel-irqchip=split \
-device intel-iommu,device-iotlb=on,intremap \
-cpu host -m 8G \
-object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \
-numa node,memdev=mem -mem-prealloc \
-smp 4,sockets=1,cores=4,threads=1 \
-device pcie-root-port,id=root.1,slot=1 \
-device pcie-root-port,id=root.2,slot=2 \
-chardev socket,id=char0,path=/tmp/vhost-user1 \
-device virtio-net-pci,netdev=mynet1,mac=aa:66:da:5f:dd:11,bus=root.1 \
-netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
-drive file=/home/rhel7.5.qcow2,format=qcow2,if=none,id=drive-virtio-blk0,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0,bus=root.2 \
-vnc :2 \
-monitor stdio \


2. Boot VM With vhost backend[2], everything works well when start up the network device in guest. Both qemu and guest work well.

[2]
-netdev tap,id=mynet1,vhost=on \
-device virtio-net-pci,netdev=mynet1,mac=aa:66:da:5f:dd:11,iommu_platform=on,ats=on,bus=root.1 \

Related full command:
/usr/libexec/qemu-kvm -name rhel7.5 -M q35,kernel-irqchip=split \
-device intel-iommu,device-iotlb=on,intremap \
-cpu host -m 8G \
-object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \
-numa node,memdev=mem -mem-prealloc \
-smp 4,sockets=1,cores=4,threads=1 \
-device pcie-root-port,id=root.1,slot=1 \
-device pcie-root-port,id=root.2,slot=2 \
-netdev tap,id=mynet1,vhost=on \
-device virtio-net-pci,netdev=mynet1,mac=aa:66:da:5f:dd:11,iommu_platform=on,ats=on,bus=root.1 \
-drive file=/home/rhel7.5.qcow2,format=qcow2,if=none,id=drive-virtio-blk0,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0,bus=root.2 \
-vnc :2 \
-monitor stdio \

Hi Pei,

I now understand why I didn't reproduce on my setup, and why it didn't reproduce 100% on yours.

The problem happens when QEMU is started on a different NUMA node than testpmd.
My setup has a single NUMA node, and in case of multiple node,
QEMU could be scheduled on the same node than testpmd and the problem wouldn't appear.

When it happens, virtqueue and virtio-net structures are reallocated on QEMU's NUMA node.
The problem is that it corrupts the IOTLB cache list, and as a side effect highlights a deadlock issue.

The fix is donein DPDK, I posted two patches to fix the issues:

commit 0911327e96516cf397334ed7b433b71f71514ee5
Author: Maxime Coquelin <maxime.coquelin>
Date:   Thu Oct 12 16:43:24 2017 +0200

    vhost: fix IOTLB on NUMA realloc
    
    In case of NUMA reallocation, virtqueue's iotlb list is broken,
    has its head changes but first iotlb entry in the list still points
    to the previous head pointer.
    
    Also, in case of reallocation, we want the IOTLB cache mempool to be
    on the new socket.
    
    This patch perform a full re-init of the IOTLB cache when mempool
    already exists, and calls the IOTLB cache init function in case
    the virtqueue is being reallocated on a new socket.
    
    Signed-off-by: Maxime Coquelin <maxime.coquelin>

commit f4275bf4e0406e8343f92adf8475e7b998ce147a
Author: Maxime Coquelin <maxime.coquelin>
Date:   Thu Oct 12 16:24:30 2017 +0200

    vhost: fix deadlock on IOTLB miss
    
    An optimization was done to only take the iotlb cache lock
    once per packet burst instead of once per IOVA translation.
    
    With this, IOTLB miss requests are sent to Qemu with the lock
    held, which can cause a deadlock if the socket buffer is full,
    and if Qemu is waiting for an IOTLB update to be done.
    
    Holding the lock is not necessary when sending an IOTLB miss
    request, as it is not manipulating the IOTLB cache list, which
    the lock protects. Let's just release it while sending the
    IOTLB miss.
    
    Signed-off-by: Maxime Coquelin <maxime.coquelin>


I posted the patches to my gitlab account so that you can have a try:

repo: https://gitlab.com/mcoquelin/dpdk-next-virtio.git
branch: https://gitlab.com/mcoquelin/dpdk-next-virtio/tree/vhost_iotlb_fixes-rc1

To trigger the issue systematically, you can use numactl to force QEMU to be scheduled on NUMA node 1:
numactl -i 1 sh qemu.sh

Thanks,
Maxime

(In reply to Maxime Coquelin from comment #10)
> Hi Pei,
> 
> I now understand why I didn't reproduce on my setup, and why it didn't
> reproduce 100% on yours.
> 
> The problem happens when QEMU is started on a different NUMA node than
> testpmd.
> My setup has a single NUMA node, and in case of multiple node,
> QEMU could be scheduled on the same node than testpmd and the problem
> wouldn't appear.
> 
> When it happens, virtqueue and virtio-net structures are reallocated on
> QEMU's NUMA node.
> The problem is that it corrupts the IOTLB cache list, and as a side effect
> highlights a deadlock issue.
> 
> The fix is donein DPDK, I posted two patches to fix the issues:
> 
> commit 0911327e96516cf397334ed7b433b71f71514ee5
> Author: Maxime Coquelin <maxime.coquelin>
> Date:   Thu Oct 12 16:43:24 2017 +0200
> 
>     vhost: fix IOTLB on NUMA realloc
>     
>     In case of NUMA reallocation, virtqueue's iotlb list is broken,
>     has its head changes but first iotlb entry in the list still points
>     to the previous head pointer.
>     
>     Also, in case of reallocation, we want the IOTLB cache mempool to be
>     on the new socket.
>     
>     This patch perform a full re-init of the IOTLB cache when mempool
>     already exists, and calls the IOTLB cache init function in case
>     the virtqueue is being reallocated on a new socket.
>     
>     Signed-off-by: Maxime Coquelin <maxime.coquelin>
> 
> commit f4275bf4e0406e8343f92adf8475e7b998ce147a
> Author: Maxime Coquelin <maxime.coquelin>
> Date:   Thu Oct 12 16:24:30 2017 +0200
> 
>     vhost: fix deadlock on IOTLB miss
>     
>     An optimization was done to only take the iotlb cache lock
>     once per packet burst instead of once per IOVA translation.
>     
>     With this, IOTLB miss requests are sent to Qemu with the lock
>     held, which can cause a deadlock if the socket buffer is full,
>     and if Qemu is waiting for an IOTLB update to be done.
>     
>     Holding the lock is not necessary when sending an IOTLB miss
>     request, as it is not manipulating the IOTLB cache list, which
>     the lock protects. Let's just release it while sending the
>     IOTLB miss.
>     
>     Signed-off-by: Maxime Coquelin <maxime.coquelin>
> 
> 
> I posted the patches to my gitlab account so that you can have a try:
> 
> repo: https://gitlab.com/mcoquelin/dpdk-next-virtio.git
> branch:
> https://gitlab.com/mcoquelin/dpdk-next-virtio/tree/vhost_iotlb_fixes-rc1

Maxime, I was trying to verify with this repo, however I hit new issue: testpmd fail boot up with below errors:

# /root/dpdk-next-virtio/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd -l 2,4,6 --socket-mem=1024,1024 -n 4 -d /root/dpdk-next-virtio/x86_64-native-linuxapp-gcc/lib/librte_pmd_vhost.so --vdev 'net_vhost0,iface=/tmp/vhost-user1' -- --portmask=3 --disable-hw-vlan -i --rxq=1 --txq=1 --nb-cores=2 --forward-mode=io
EAL: Detected 20 lcore(s)
ERROR: failed to parse device "net_vhost0"
EAL: Unable to parse device 'net_vhost0,iface=/tmp/vhost-user1'
PANIC in main():
Cannot init EAL
5: [/root/dpdk-next-virtio/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd() [0x420d07]]
4: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f889551a135]]
3: [/root/dpdk-next-virtio/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd(main+0x972) [0x420ca2]]
2: [/root/dpdk-next-virtio/x86_64-native-linuxapp-gcc/lib/librte_eal.so.6.1(__rte_panic+0xba) [0x7f889651315a]]
1: [/root/dpdk-next-virtio/x86_64-native-linuxapp-gcc/lib/librte_eal.so.6.1(rte_dump_stack+0x1a) [0x7f889651d75a]]
Aborted

Best Regards,
Pei

> To trigger the issue systematically, you can use numactl to force QEMU to be
> scheduled on NUMA node 1:
> numactl -i 1 sh qemu.sh
> 
> Thanks,
> Maxime

Hi Pei,

I just reproduce your issue too.
It seems something is broken in current upstream master, on which my branch is based.

In the mean time, you can try with shared libs, or I can rebase my two patches on top of the vhost_iotlb_v2 branch you used initially.

Cheers,
Maxime

(In reply to Maxime Coquelin from comment #12)
> In the mean time, you can try with shared libs.

Sorry, I meant without shared libs.

Maxime

(In reply to Maxime Coquelin from comment #12)
> Hi Pei,
> 
> I just reproduce your issue too.
> It seems something is broken in current upstream master, on which my branch
> is based.
> 
> In the mean time, you can try with shared libs, or I can rebase my two
> patches on top of the vhost_iotlb_v2 branch you used initially.

I tried without shared libs as[1], then both qemu and VM work well, no hang any more. So the issue has gone with your fix.

[1]Keep CONFIG_RTE_BUILD_SHARED_LIB in x86_64-native-linuxapp-gcc/.config as default value n.


Thanks,
Pei

> Cheers,
> Maxime

Fixes applied upstream, part of DPDK v17.11-rc1 tag.

Since this bug is fixed in dpdk, so move component of this from qemu to dpdk.

The fix being in DPDK's vhost-user library, moving to Openvswitch component.
The fix is released in DPDK v17.11 LTS.

==Verification==
kernel-3.10.0-855.el7.x86_64
qemu-kvm-rhev-2.10.0-21.el7.x86_64
libvirt-3.9.0-13.el7.x86_64
dpdk-17.11-7.el7.x86_64
openvswitch-2.9.0-1.el7fdb.x86_64
microcode-20180108.tgz

Steps:

1. Boot OVS with vIOMMU and dpdkvhostuserclient

# ovs-vsctl show
d73b38bd-883d-471a-8bd5-5dd624885f01
    Bridge "ovsbr1"
        Port "dpdk1"
            Interface "dpdk1"
                type: dpdk
                options: {dpdk-devargs="0000:81:00.1", n_rxq="2"}
        Port "vhost-user1"
            Interface "vhost-user1"
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/vhostuser1.sock"}
        Port "ovsbr1"
            Interface "ovsbr1"
                type: internal
    Bridge "ovsbr0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "dpdk0"
            Interface "dpdk0"
                type: dpdk
                options: {dpdk-devargs="0000:81:00.0", n_rxq="2"}
        Port "vhost-user0"
            Interface "vhost-user0"
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/vhostuser0.sock"}


2. Boot VM with vhostuser

    <interface type='vhostuser'>
      <mac address='88:66:da:5f:dd:02'/>
      <source type='unix' path='/tmp/vhostuser0.sock' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2' rx_queue_size='1024' iommu='on' ats='on'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='88:66:da:5f:dd:03'/>
      <source type='unix' path='/tmp/vhostuser1.sock' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2' rx_queue_size='1024' iommu='on' ats='on'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>

3. In VM, start up network

# ifconfig eth1 up
# ifconfig eth2 up

4. Reboot VM

# reboot

5. Repeat step3~4 10 times, both qemu and guest work well.  


So this bug has been fixed very well. Move status of this bug to 'VERIFIED'.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0550