The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.
Bug 2160718 - The guest start failed when running ovs dpdk pvp 1 queue viommu case on kernel 5.14.0-230.mergert.el9.x86_64
Summary: The guest start failed when running ovs dpdk pvp 1 queue viommu case on kerne...
Keywords:
Status: CLOSED DUPLICATE of bug 2155173
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: openvswitch2.17
Version: FDP 22.L
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: ---
Assignee: Maxime Coquelin
QA Contact: ovs-qe
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-01-13 13:09 UTC by liting
Modified: 2023-01-19 20:01 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-01-19 20:01:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FD-2605 0 None None None 2023-01-13 13:10:02 UTC

Description liting 2023-01-13 13:09:31 UTC
Description of problem:


Version-Release number of selected component (if applicable):
[root@dell-per730-52 ~]# uname -r
5.14.0-230.mergert.el9.x86_64
[root@dell-per730-52 ~]# rpm -qa|grep openvs
kernel-kernel-networking-openvswitch-common-2.0-279.noarch
kernel-kernel-networking-openvswitch-perf-1.0-371.noarch
openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch
openvswitch2.17-2.17.0-55.el9fdp.x86_64


How reproducible:


Steps to Reproduce:
Run ovs dpdk pvp 1queue viommu case
1. Build ovs dpdk pvp topo
[root@dell-per730-52 ~]# ovs-vsctl show
d44eedd4-336a-4f9a-9b2e-ea4e8c83bd10
    Bridge ovsbr0
        datapath_type: netdev
        Port dpdk1
            Interface dpdk1
                type: dpdk
                options: {dpdk-devargs="0000:07:00.1", n_rxq="1", n_rxq_desc="1024", n_txq_desc="1024"}
        Port vhost0
            Interface vhost0
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/vhostuser/vhost0"}
        Port vhost1
            Interface vhost1
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/vhostuser/vhost1"}
        Port ovsbr0
            Interface ovsbr0
                type: internal
        Port dpdk0
            Interface dpdk0
                type: dpdk
                options: {dpdk-devargs="0000:07:00.0", n_rxq="1", n_rxq_desc="1024", n_txq_desc="1024"}
    ovs_version: "2.17.4"
[root@dell-per730-52 ~]# ovs-vsctl get Open_vSwitch . other_config
{dpdk-init="true", dpdk-lcore-mask="0x1", dpdk-socket-mem="4096", pmd-cpu-mask="40000004000000", userspace-tso-enable="false", vhost-iommu-support="true"}

2. start guest with following xml
<domain type='kvm'>
  <name>g1</name>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <locked/>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='30'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <emulatorpin cpuset='0,28'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/g1.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:01:02:03'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='00:de:ad:00:00:01'/>
      <source type='unix' path='/tmp/vhostuser/vhost0' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' queues='1' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'>
        <host mrg_rxbuf='off'/>
      </driver>
      <address type='pci' domain='0x0000' bus='0x3' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='00:de:ad:00:00:02'/>
      <source type='unix' path='/tmp/vhostuser/vhost1' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' queues='1' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'>
        <host mrg_rxbuf='off'/>
      </driver>
      <address type='pci' domain='0x0000' bus='0x4' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
    </iommu>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'/>
</domain>


Actual results:
Start guest failed. So the 1 queue viommu case run failed.

Expected results:
The 1 queue viommu case can work well.

Additional info:
https://beaker.engineering.redhat.com/jobs/7424884

Comment 1 liting 2023-01-13 13:16:18 UTC
libvirt log:
[root@dell-per730-52 ~]# cat /var/log/libvirt/qemu/g1.log
2023-01-13 13:10:44.059+0000: starting up libvirt version: 8.10.0, package: 2.el9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2022-12-06-04:27:41, ), qemu version: 7.2.0qemu-kvm-7.2.0-2.el9, kernel: 5.14.0-230.mergert.el9.x86_64, hostname: dell-per730-52.rhts.eng.pek2.redhat.com
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-g1 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-g1/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-g1/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-g1/.config \
/usr/libexec/qemu-kvm \
-name guest=g1,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-g1/master-key.aes"}' \
-machine pc-q35-rhel9.2.0,usb=off,vmport=off,kernel_irqchip=split,dump-guest-core=off \
-accel kvm \
-cpu host,migratable=on,tsc-deadline=on,pmu=off \
-m 8192 \
-overcommit mem-lock=on \
-smp 3,sockets=3,cores=1,threads=1 \
-object '{"qom-type":"thread-context","id":"tc-ram-node0","node-affinity":[0]}' \
-object '{"qom-type":"memory-backend-file","id":"ram-node0","mem-path":"/dev/hugepages/libvirt/qemu/1-g1","share":true,"prealloc":true,"size":8589934592,"host-nodes":[0],"policy":"bind","prealloc-context":"tc-ram-node0"}' \
-numa node,nodeid=0,cpus=0-2,memdev=ram-node0 \
-uuid 36095c8c-6865-41b1-9c0d-c3a6731ac6aa \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"intel-iommu","id":"iommu0","intremap":"on","caching-mode":true,"device-iotlb":true}' \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x3"}' \
-device '{"driver":"pcie-root-port","port":8,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x4"}' \
-device '{"driver":"pcie-root-port","port":9,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x5"}' \
-device '{"driver":"pcie-root-port","port":10,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x6"}' \
-device '{"driver":"pcie-root-port","port":11,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x7"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/g1.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.1","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1}' \
-netdev '{"type":"tap","fd":"33","vhost":true,"vhostfd":"35","id":"hostnet0"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:01:02:03","bus":"pci.2","addr":"0x0"}' \
-chardev socket,id=charnet1,path=/tmp/vhostuser/vhost0,server=on \
-netdev '{"type":"vhost-user","chardev":"charnet1","id":"hostnet1"}' \
-device '{"driver":"virtio-net-pci","iommu_platform":true,"ats":true,"mrg_rxbuf":false,"rx_queue_size":1024,"tx_queue_size":1024,"netdev":"hostnet1","id":"net1","mac":"00:de:ad:00:00:01","bus":"pci.3","addr":"0x0"}' \
-chardev socket,id=charnet2,path=/tmp/vhostuser/vhost1,server=on \
-netdev '{"type":"vhost-user","chardev":"charnet2","id":"hostnet2"}' \
-device '{"driver":"virtio-net-pci","iommu_platform":true,"ats":true,"mrg_rxbuf":false,"rx_queue_size":1024,"tx_queue_size":1024,"netdev":"hostnet2","id":"net2","mac":"00:de:ad:00:00:02","bus":"pci.4","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:0,audiodev=audio1 \
-device '{"driver":"cirrus-vga","id":"video0","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.6","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2023-01-13T13:10:44.130006Z qemu-kvm: -chardev socket,id=charnet1,path=/tmp/vhostuser/vhost0,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhostuser/vhost0,server=on
2023-01-13T13:10:44.644338Z qemu-kvm: -chardev socket,id=charnet2,path=/tmp/vhostuser/vhost1,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhostuser/vhost1,server=on
char device redirected to /dev/pts/2 (label charserial0)
2023-01-13T13:10:47.444502Z qemu-kvm: -device {"driver":"cirrus-vga","id":"video0","bus":"pci.5","addr":"0x0"}: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
2023-01-13T13:10:57.093391Z qemu-kvm: Received unexpected msg type. Expected 22 received 40
2023-01-13T13:10:57.093414Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:10:57.093463Z qemu-kvm: Received unexpected msg type. Expected 40 received 22
2023-01-13T13:10:57.104339Z qemu-kvm: Received unexpected msg type. Expected 22 received 11
2023-01-13T13:10:57.104349Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:10:57.104583Z qemu-kvm: Received unexpected msg type. Expected 11 received 22
2023-01-13T13:10:57.104592Z qemu-kvm: vhost VQ 1 ring restore failed: -71: Protocol error (71)
2023-01-13T13:10:57.108169Z qemu-kvm: unable to start vhost net: 71: falling back on userspace virtio
2023-01-13T13:10:57.208006Z qemu-kvm: Received unexpected msg type. Expected 22 received 40
2023-01-13T13:10:57.208020Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:10:57.208071Z qemu-kvm: Received unexpected msg type. Expected 40 received 22
2023-01-13T13:10:57.208321Z qemu-kvm: Received unexpected msg type. Expected 22 received 11
2023-01-13T13:10:57.208330Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:10:57.208538Z qemu-kvm: Received unexpected msg type. Expected 11 received 22
2023-01-13T13:10:57.208546Z qemu-kvm: vhost VQ 1 ring restore failed: -71: Protocol error (71)
2023-01-13T13:10:57.208607Z qemu-kvm: Received unexpected msg type. Expected 22 received 11
2023-01-13T13:10:57.208615Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:10:57.208641Z qemu-kvm: Received unexpected msg type. Expected 11 received 22
2023-01-13T13:10:57.208649Z qemu-kvm: vhost VQ 0 ring restore failed: -71: Protocol error (71)
2023-01-13T13:10:57.212170Z qemu-kvm: unable to start vhost net: 71: falling back on userspace virtio
2023-01-13T13:12:26.201845Z qemu-kvm: Received unexpected msg type. Expected 22 received 40
2023-01-13T13:12:26.201867Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:12:26.201916Z qemu-kvm: Received unexpected msg type. Expected 40 received 22
2023-01-13T13:12:26.202204Z qemu-kvm: Received unexpected msg type. Expected 22 received 11
2023-01-13T13:12:26.202213Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:12:26.202445Z qemu-kvm: Received unexpected msg type. Expected 11 received 22
2023-01-13T13:12:26.202454Z qemu-kvm: vhost VQ 1 ring restore failed: -71: Protocol error (71)

Comment 2 Jiying Qiu 2023-01-18 02:11:05 UTC
I met the same issue. 
https://beaker.engineering.redhat.com/recipes/13233855#task154894511 

BTW- 2q works well.

Comment 6 Maxime Coquelin 2023-01-19 20:01:58 UTC
This bug is a duplicate of Bz2155173.
The issue is on QEMU side, not in OVS.

Closing as duplicate.

*** This bug has been marked as a duplicate of bug 2155173 ***


Note You need to log in before you can comment on or make changes to this bug.