Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1921462

Summary: i40e/ice driver: ipv6 ping failed when sr-iov vf add to ovs bridge
Product: Red Hat Enterprise Linux Fast Datapath Reporter: liting <tli>
Component: openvswitch2.13Assignee: David Marchand <dmarchan>
Status: CLOSED MIGRATED QA Contact: liting <tli>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: FDP 21.ACC: cgoncalves, ctrautma, fleitner, jhsiao, ralongi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-03-12 10:19:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description liting 2021-01-28 02:48:30 UTC 
Description of problem:
i40e driver: ipv6 ping failed when sr-iov vf add to ovs bridge

Version-Release number of selected component (if applicable):
[root@netqe22 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.3 (Ootpa)
[root@netqe22 ~]# uname -a
Linux netqe22.knqe.lab.eng.bos.redhat.com 4.18.0-240.el8.x86_64 #1 SMP Wed Sep 23 05:13:10 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@netqe22 ~]# rpm -qa|grep dpdk
dpdk-19.11.3-1.el8.x86_64
dpdk-tools-19.11.3-1.el8.x86_64

[root@netqe22 ~]# rpm -qa|grep openv
openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch
openvswitch2.13-2.13.0-77.el8fdp.x86_64

[root@netqe22 ~]# ethtool -i enp3s0f1
driver: i40e
version: 2.8.20-k
firmware-version: 6.01 0x800035cf 1.1747.0
expansion-rom-version: 
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

How reproducible:


Steps to Reproduce:
Netqe22 i40e nic connect with Netqe32 i40e nic directly.

Two server build same topo. On netqe22
1. create two vf for pf enp3s0f0

2.Bind vf 1 to dpdk, and add dpdk0 to ovs bridge.
 /usr/share/dpdk/usertools/dpdk-devbind.py -b vfio-pci 0000:03:02.1
 systemctl restart openvswitch
 ovs-vsctl set Open_vSwitch . 'other_config={}'
 ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
 ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=1024,1024
 ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x500000500000
 ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
 ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk type=dpdk options:dpdk-devargs=0000:03:02.1
 ovs-vsctl add-port ovsbr0 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient -- set Interface dpdkvhostuserclient0 options:vhost-server-path=/tmp/dpdkvhostuserclient0

3.Configure the vf 1 mac same with guest's mac
ip link set enp3s0f0 vf 1 mac $guest_mac

4. Inside guest, configure ipv4 and ipv6 address for eth0
ip addr add 20.0.0.2/24 dev eth0
ip addr add 2001:5c0:9168::2/24 dev eth0

5. On netqe32 system, run above similar command on it. 

[root@netqe22 ~]# ovs-vsctl show
0f33bdd1-3a4d-4b05-ba46-55413d87c26d
    Bridge ovsbr0
        datapath_type: netdev
        Port ovsbr0
            Interface ovsbr0
                type: internal
        Port dpdk0
            Interface dpdk0
                type: dpdk
                options: {dpdk-devargs="0000:03:02.1"}
        Port dpdkvhostuserclient0
            Interface dpdkvhostuserclient0
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/dpdkvhostuserclient0"}
    ovs_version: "2.13.2"
6. On netqe32 guest, ping the ip on netqe22 guest.


Actual results:
On netqe32, ipv4 ping successfully, ipv6 ping failed. When configure vf 1 trust on, ipv6 still ping failed.

Expected results:
ipv6 can ping successfully.

Additional info:
https://beaker.engineering.redhat.com/jobs/5035715
Comment 1 liting 2021-08-06 14:35:34 UTC 
The issue still exist on openvswitch2.15-2.15.0-26.
[root@netqe22 ~]# uname -r
4.18.0-305.el8.x86_64
[root@netqe22 ~]# rpm -qa|grep openvswitch
openvswitch2.15-2.15.0-26.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch
beaker job link.
https://beaker.engineering.redhat.com/jobs/5677369
Comment 2 liting 2021-08-06 14:38:41 UTC 
The issue also exist on ice card.
https://beaker.engineering.redhat.com/jobs/5613952
Comment 3 liting 2021-08-09 06:29:58 UTC 
After disable spoof check, and turn trust on. The ipv6 ping successfully.
Comment 4 liting 2022-05-23 09:39:49 UTC 
For ice card. it also exist this issue. ipv6 ping failed
https://beaker.engineering.redhat.com/jobs/6644207
Comment 5 liting 2022-06-13 10:17:05 UTC 
For fdp22e, rhel9.0, ice card. this issue still exist
https://beaker.engineering.redhat.com/jobs/6713629
Comment 6 liting 2022-07-15 02:54:05 UTC 
for fdp22F, ice card, this issue still exist
https://beaker.engineering.redhat.com/jobs/6815954
Comment 7 liting 2022-08-30 12:24:32 UTC 
for fdp22G, ice card, this issue still exist.
https://beaker.engineering.redhat.com/jobs/6956356
Comment 8 Stefan Assmann 2022-09-12 13:57:58 UTC 
What happens if you use the iavf driver from the kernel instead of dpdk?
Comment 9 liting 2022-10-09 10:44:16 UTC 
(In reply to Stefan Assmann from comment #8)
> What happens if you use the iavf driver from the kernel instead of dpdk?

I try to use iavf driver to run the same cases with kernel, the ipv6 ping succcessfully. ipv6 ping failed only exist in dpdk case.
Comment 10 liting 2022-11-11 06:48:03 UTC 
For fdp22k, this issue still exist.
i40e:
https://beaker.engineering.redhat.com/jobs/7210502
Comment 11 liting 2022-12-08 07:10:34 UTC 
For fdp22.L, ice still has this issue.
https://beaker.engineering.redhat.com/jobs/7316969
https://beaker.engineering.redhat.com/jobs/7316975
Comment 12 liting 2023-02-15 07:14:49 UTC 
For fdp23.A, ice still has this issue
https://beaker.engineering.redhat.com/jobs/7529038
Comment 13 liting 2023-04-10 03:41:15 UTC 
For fdp23.C, i40e has no this issue, but ice still has this issue. 
ice rhel9.2 job:
https://beaker.engineering.redhat.com/jobs/7719135
ice rhel8.6 job:
https://beaker.engineering.redhat.com/jobs/7715761
Comment 18 liting 2023-10-18 01:12:13 UTC 
It still happened with ovs3.2/dpdk-22.11-4.el9 on rhel9.2 on ice card.
https://beaker.engineering.redhat.com/jobs/8440484
Comment 19 liting 2023-10-19 06:26:43 UTC 
It has no this issue with  openvswitch3.1-3.1.0-54.el9fdp/dpdk-22.11-4.el9 on i40e card.
https://beaker.engineering.redhat.com/jobs/8449516
It has no this issue with  openvswitch3.2-3.2.0-17.el9fdp/dpdk-22.11-4.el9 on i40e card.
https://beaker.engineering.redhat.com/jobs/8446277
Comment 20 liting 2023-11-13 09:47:57 UTC 
For ice driver, qinq case also ping ipv6 failed.
https://beaker.engineering.redhat.com/jobs/8547980
Comment 21 liting 2023-11-17 03:21:43 UTC 
For this ipv6 ping failed issue, I found it can ping ipv6 address successfully when vf configure trust on. But ipv6 ping failed when vf configure trust off.
For ice driver, qinq case(SW Inner / HW outer QinQ) aslo ping ipv6 failed when vf configure trust off. Test steps:
ice card on wsfd-advnetlab10.anl.lab.eng.bos.redhat.com <----directly connect----> ice card on wsfd-advnetlab11.anl.lab.eng.bos.redhat.com 
wsfd-advnetlab10.anl.lab.eng.bos.redhat.com:
#create two vfs for pf ens1f0
5: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 40:a6:b7:96:aa:bc brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:60:2f:ed:a6:64 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether 52:54:00:11:8f:ea brd ff:ff:ff:ff:ff:ff, vlan 4, vlan protocol 802.1ad, spoof checking on, link-state auto, trust off
#configure the vf 1 mac same with the guest's mac
 ip link set ens1f0 vf 1 mac 52:54:00:11:8f:ea
 ip link set ens1f0 vf 1 vlan 4 proto 802.1ad
 driverctl -v set-override 0000:3b:01.1 vfio-pci
 ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=1024,1024
 ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
 ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x44000044000
 ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
 ovs-vsctl add-br ovsbr1 -- set bridge ovsbr1 datapath_type=netdev
 ovs-vsctl add-port ovsbr1 dpdk0 -- set Interface dpdk0 type=dpdk type=dpdk options:dpdk-devargs=0000:3b:01.1
 ovs-vsctl add-port ovsbr0 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient -- set Interface dpdkvhostuserclient0 options:vhost-server-path=/tmp/dpdkvhostuserclient0
 ovs-vsctl add-port ovsbr0 patch-ovs-0 -- set Interface patch-ovs-0 type=patch options:peer=patch-ovs-1
 ovs-vsctl add-port ovsbr1 patch-ovs-1 -- set Interface patch-ovs-1 type=patch options:peer=patch-ovs-0
 ip link set ovsbr0 up
 ip link set ovsbr1 up

 inside guest:
 ip link add link enp4s0 name enp4s0.3 type vlan id 3 proto 802.1q
 ip link set enp4s0.3 up
 ip addr add 20.0.0.1/24 dev enp4s0.3
 ip addr add 2001:5c0:9168::1/24 dev enp4s0.3

wsfd-advnetlab11.anl.lab.eng.bos.redhat.com:
create two vfs for pf ens1f0.
 6: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 40:a6:b7:96:bf:b8 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:60:2f:fb:15:53 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether 52:54:00:11:8f:e9 brd ff:ff:ff:ff:ff:ff, vlan 4, vlan protocol 802.1ad, spoof checking on, link-state auto, trust off

 #configure the vf 1 mac same with the guest's mac
 ip link set ens1f0 vf 1 mac 52:54:00:11:8f:e9
 ip link set ens1f0 vf 1 vlan 4 proto 802.1ad
 driverctl -v set-override 0000:3b:01.1 vfio-pci
 ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=1024,1024
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x44000044000
ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
ovs-vsctl add-br ovsbr1 -- set bridge ovsbr1 datapath_type=netdev
ovs-vsctl add-port ovsbr1 dpdk0 -- set Interface dpdk0 type=dpdk type=dpdk options:dpdk-devargs=0000:3b:01.1
ovs-vsctl add-port ovsbr0 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient -- set Interface dpdkvhostuserclient0 options:vhost-server-path=/tmp/dpdkvhostuserclient0
ovs-vsctl add-port ovsbr0 patch-ovs-0 -- set Interface patch-ovs-0 type=patch options:peer=patch-ovs-1
ovs-vsctl add-port ovsbr1 patch-ovs-1 -- set Interface patch-ovs-1 type=patch options:peer=patch-ovs-0
ip link set ovsbr0 up
ip link set ovsbr1 up

inside guest:
 ip link add link enp4s0 name enp4s0.3 type vlan id 3 proto 802.1q
 ip link set enp4s0.3 up
 ip addr add 20.0.0.2/24 dev enp4s0.3
 ip addr add 2001:5c0:9168::2/24 dev enp4s0.3

 ping 20.0.0.1 -c 10   ----successfully
 ping6 2001:5c0:9168::1   -------failed


For i40e card, qinq case(Full SW QinQ) also exist this issue, when vf configure trust off,the ipv6 ping failed.
Comment 22 Carlos Goncalves 2023-11-20 14:23:42 UTC 
Does it work with kernel VLANs instead of OVS/DPDK for SW Inner?
Comment 23 liting 2023-11-27 02:22:01 UTC 
(In reply to Carlos Goncalves from comment #22)
> Does it work with kernel VLANs instead of OVS/DPDK for SW Inner?

I have not test kernel vlans case, I will test it later.
Comment 24 liting 2023-11-30 10:04:01 UTC 
The issue also exist on bnxt_en driver card. 
job: https://beaker.engineering.redhat.com/jobs/8615536
netqe22 bnxt_en card <---directly connect--> netqe32 i40e driver card
netqe22 bnxt side:
echo 2 > /sys/devices/pci0000:80/0000:80:02.0/0000:82:00.0/sriov_numvfs
ip link set enp130s0f0np0 vf 1 trust on
ip link set enp130s0f0np0 vf 1 vlan 3
driverctl -v set-override 0000:82:02.1 vfio-pci
systemctl restart openvswitch
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=1024,1024
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xa00000a00000
ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk type=dpdk options:dpdk-devargs=0000:82:02.1
ovs-vsctl add-port ovsbr0 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient -- set Interface dpdkvhostuserclient0 options:vhost-server-path=/tmp/dpdkvhostuserclient0
[root@netqe22 no-bond]# virsh dumpxml master3
<domain type='kvm' id='1'>
  <name>master3</name>
  <uuid>37425e76-af6a-44a6-aba0-73434afe34c0</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='3'/>
    <vcpupin vcpu='1' cpuset='5'/>
    <vcpupin vcpu='2' cpuset='29'/>
    <emulatorpin cpuset='3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='6291456' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/master3.qcow2' index='1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='vhostuser'>
      <mac address='52:54:00:11:8f:ea'/>
      <source type='unix' path='/tmp/dpdkvhostuserclient0' mode='server'/>
      <target dev='dpdkvhostuserclient0'/>
      <model type='virtio'/>
      <driver name='vhost'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:bb:63:7b'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'>
      <alias name='watchdog0'/>
    </watchdog>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
      <alias name='iommu0'/>
    </iommu>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c257,c997</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c257,c997</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+983</label>
    <imagelabel>+107:+983</imagelabel>
  </seclabel>
</domain>

inside guest:
ip addr add 20.0.0.2/24 dev enp4s0
ip addr add 2001:5c0:9168::1/64 dev enp4s0


netqe32 i40e side:
echo 2 > /sys/devices/pci0000:5d/0000:5d:00.0/0000:5e:00.0/sriov_numvfs
ip link set ens3f0 vf 1 trust on
ip link set ens3f0 vf 1 vlan 3
driverctl -v set-override 0000:5e:02.1 vfio-pci
systemctl restart openvswitch
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=1024,1024
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x440000044000
ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk type=dpdk options:dpdk-devargs=0000:5e:02.1
ovs-vsctl add-port ovsbr0 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient -- set Interface dpdkvhostuserclient0 options:vhost-server-path=/tmp/dpdkvhostuserclient0
ip link set ovsbr0 up

[root@netqe32 ~]# virsh dumpxml master3
<domain type='kvm' id='1'>
  <name>master3</name>
  <uuid>37425e76-af6a-44a6-aba0-73434afe34c0</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='4'/>
    <vcpupin vcpu='2' cpuset='32'/>
    <emulatorpin cpuset='2'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='6291456' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/master3.qcow2' index='1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='vhostuser'>
      <mac address='52:54:00:11:8f:e9'/>
      <source type='unix' path='/tmp/dpdkvhostuserclient0' mode='server'/>
      <target dev='dpdkvhostuserclient0'/>
      <model type='virtio'/>
      <driver name='vhost'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:bb:63:7b'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'>
      <alias name='watchdog0'/>
    </watchdog>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
      <alias name='iommu0'/>
    </iommu>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c502,c908</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c502,c908</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+983</label>
    <imagelabel>+107:+983</imagelabel>
  </seclabel>
</domain>

inside guest:
ip addr add 20.0.0.2/24 dev enp4s0
ip addr add 2001:5c0:9168::2/64 dev enp4s0


When configure vf trust on.
Inside guest of netqe32, ping 20.0.0.1 successfully. Ping 2001:5c0:9168::1 failed.
And inside guest of netqe22, ping 20.0.0.2 successfully, ping 2001:5c0:9168::6 successfully. 
After netqe22 ping netqe32 successfully, and then netqe32 can ping 2001:5c0:9168::1 successully.
When configure vf trust off,
Inside guest of netqe32, ping 20.0.0.1 successfully. Ping 2001:5c0:9168::1 failed.
Inside guest of netqe22, ping 20.0.0.2 successfully, ping 2001:5c0:9168::6 failed.
Comment 25 liting 2023-11-30 12:38:53 UTC 
(In reply to Carlos Goncalves from comment #22)
> Does it work with kernel VLANs instead of OVS/DPDK for SW Inner?

I run the ovs dpdk qinq kernel case for SW Inner, ipv6 ping successfully when vf configure trust off. So the issue only occur on OVS/DPDK case.
730-52 i40e card <---directly connect--> 730-53 i40e card
730-52 side:
[root@dell-per730-52 ~]# ip a
6: enp7s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:ad:bc:e8 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:60:2f:c4:c4:12 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether 52:54:00:11:8f:ea brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off

[root@dell-per730-52 ~]# ovs-vsctl show
54e496c6-7526-47d3-90c0-305d60a2be9a
    Bridge ovsbr1
        Port patch-ovs-1
            Interface patch-ovs-1
                type: patch
                options: {peer=patch-ovs-0}
        Port enp7s0f0v1
            Interface enp7s0f0v1
        Port ovsbr1
            Interface ovsbr1
                type: internal
    Bridge ovsbr0
        Port tap0
            tag: 4
            Interface tap0
        Port ovsbr0
            Interface ovsbr0
                type: internal
        Port patch-ovs-0
            Interface patch-ovs-0
                type: patch
                options: {peer=patch-ovs-1}
    ovs_version: "2.17.9"

with following xml start guest
[root@dell-per730-52 ~]# virsh dumpxml master4
<domain type='kvm' id='1'>
  <name>master4</name>
  <uuid>37425e76-af6a-44a6-aba0-73434afe34c2</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB' nodeset='0'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <emulatorpin cpuset='1'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/master4.qcow2' index='1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:11:8f:ea'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='f54fd445-25b0-4231-8ea7-b83f2a1f8df2'/>
      </virtualport>
      <target dev='tap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:bb:63:7b'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-master4/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'>
      <alias name='watchdog0'/>
    </watchdog>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c107,c748</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c107,c748</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+983</label>
    <imagelabel>+107:+983</imagelabel>
  </seclabel>
</domain>

inside guest:
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:bb:63:7b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.138/24 brd 192.168.122.255 scope global dynamic noprefixroute enp0s3
       valid_lft 2985sec preferred_lft 2985sec
    inet6 fe80::a061:c39c:a5f3:16ff/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:11:8f:ea brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.1/24 scope global enp0s9
       valid_lft forever preferred_lft forever
    inet6 2001:5c0:9168::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::a2cf:7cde:10cf:9010/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever


dell730-53 side:
[root@dell-per730-53 ~]# ip link show
6: enp4s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:ad:bf:c4 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:60:2f:46:cf:4c brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether 52:54:00:11:8f:e9 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off

[root@dell-per730-53 ~]# ovs-vsctl show
f27bbe51-755d-4d95-be10-b32aff047a62
    Bridge ovsbr1
        Port ovsbr1
            Interface ovsbr1
                type: internal
        Port patch-ovs-1
            Interface patch-ovs-1
                type: patch
                options: {peer=patch-ovs-0}
        Port enp4s0f0v1
            Interface enp4s0f0v1
    Bridge ovsbr0
        Port tap0
            tag: 4
            Interface tap0
        Port ovsbr0
            Interface ovsbr0
                type: internal
        Port patch-ovs-0
            Interface patch-ovs-0
                type: patch
                options: {peer=patch-ovs-1}
    ovs_version: "2.17.9"

start guest with following xml
[root@dell-per730-53 ~]# virsh dumpxml master4
<domain type='kvm' id='1'>
  <name>master4</name>
  <uuid>37425e76-af6a-44a6-aba0-73434afe34c2</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB' nodeset='0'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <emulatorpin cpuset='1'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/master4.qcow2' index='1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:11:8f:e9'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='f54fd445-25b0-4231-8ea7-b83f2a1f8df2'/>
      </virtualport>
      <target dev='tap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:bb:63:7b'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-master4/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'>
      <alias name='watchdog0'/>
    </watchdog>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c297,c588</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c297,c588</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+983</label>
    <imagelabel>+107:+983</imagelabel>
  </seclabel>
</domain>


inside guest:
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:bb:63:7b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.138/24 brd 192.168.122.255 scope global dynamic noprefixroute enp0s3
       valid_lft 2694sec preferred_lft 2694sec
    inet6 fe80::d99:29d1:35b4:dacb/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:11:8f:e9 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.2/24 scope global enp0s9
       valid_lft forever preferred_lft forever
    inet6 2001:5c0:9168::2/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::8725:1aa6:b00e:1e7e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@localhost ~]# ping 20.0.0.1 -c 1
PING 20.0.0.1 (20.0.0.1) 56(84) bytes of data.
64 bytes from 20.0.0.1: icmp_seq=1 ttl=64 time=0.599 ms

--- 20.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.599/0.599/0.599/0.000 ms

[root@localhost ~]# ping6 2001:5c0:9168::1 -c 1
PING 2001:5c0:9168::1(2001:5c0:9168::1) 56 data bytes
64 bytes from 2001:5c0:9168::1: icmp_seq=1 ttl=64 time=0.575 ms

--- 2001:5c0:9168::1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.575/0.575/0.575/0.000 ms
Comment 26 liting 2023-11-30 13:24:16 UTC 
Sorry, the comment#25 is just test vlan kernel case. I forgot configure the vlan interface inside guest. When I create vlan interface inside guest, and it should be qinq case  SW Inner.
For ovs kernel qinq case for SW Inner.
 ipv6 ping failed when configure vf trust off. 
 ipv6 ping successfully when configure vf trust on.

So this issue also exist on ovs kernel qinq case for SW Inner.

dell730-52 guest:
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:bb:63:7b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.138/24 brd 192.168.122.255 scope global dynamic noprefixroute enp0s3
       valid_lft 2942sec preferred_lft 2942sec
    inet6 fe80::a061:c39c:a5f3:16ff/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:11:8f:ea brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a2cf:7cde:10cf:9010/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: enp0s9.3@enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:11:8f:ea brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.1/24 scope global enp0s9.3
       valid_lft forever preferred_lft forever
    inet6 2001:5c0:9168::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe11:8fea/64 scope link 
       valid_lft forever preferred_lft forever


dell730-53 guest:
 [root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:bb:63:7b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.138/24 brd 192.168.122.255 scope global dynamic noprefixroute enp0s3
       valid_lft 2970sec preferred_lft 2970sec
    inet6 fe80::d99:29d1:35b4:dacb/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:11:8f:e9 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8725:1aa6:b00e:1e7e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: enp0s9.3@enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:11:8f:e9 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.2/24 scope global enp0s9.3
       valid_lft forever preferred_lft forever
    inet6 2001:5c0:9168::2/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe11:8fe9/64 scope link 
       valid_lft forever preferred_lft forever

[root@localhost ~]# ping6 2001:5c0:9168::1
PING 2001:5c0:9168::1(2001:5c0:9168::1) 56 data bytes
From 2001:5c0:9168::2 icmp_seq=1 Destination unreachable: Address unreachable
From 2001:5c0:9168::2 icmp_seq=2 Destination unreachable: Address unreachable
From 2001:5c0:9168::2 icmp_seq=3 Destination unreachable: Address unreachable
From 2001:5c0:9168::2 icmp_seq=4 Destination unreachable: Address unreachable
From 2001:5c0:9168::2 icmp_seq=5 Destination unreachable: Address unreachable
From 2001:5c0:9168::2 icmp_seq=6 Destination unreachable: Address unreachable
Comment 27 liting 2023-12-04 02:46:44 UTC 
update the ovs kernel qinq SW Inner case. 
For vf configure trust on, this IPv6 does not always ping successfully.
https://beaker.engineering.redhat.com/jobs/8628754
Comment 28 liting 2023-12-04 03:55:29 UTC 
For rhel8.10, when vf configure trust on, ipv6 cannot ping successfully on i40e driver.
https://beaker.engineering.redhat.com/jobs/8628880
Comment 29 liting 2024-01-04 05:54:32 UTC 
For rhel9.2, 740-57 ice <--connect--> 740-73 i40e env, ping ipv6 address not always successfully
https://beaker.engineering.redhat.com/jobs/8752006
Comment 30 liting 2024-01-04 08:16:38 UTC 
For rhel9.2, 730-52 i40e <--connect--> 730-53 i40e env, ping ipv6 address not always successfully
https://beaker.engineering.redhat.com/jobs/8752408
Comment 31 liting 2024-01-05 01:44:17 UTC 
For rhel9.2 and rhel8.6, netqe45 ice <--connect--> netqe46 ice env, ping ipv6 address not always successfully.
rhel9.2 ovs2.17:
https://beaker.engineering.redhat.com/jobs/8753517
rhel9.2 ovs3.2:
https://beaker.engineering.redhat.com/jobs/8753511
https://beaker.engineering.redhat.com/jobs/8753510
rhel9.2 ovs3.1
https://beaker.engineering.redhat.com/jobs/8752597
rhel8.6 ovs2.17
https://beaker.engineering.redhat.com/jobs/8753502
Comment 32 liting 2024-01-05 01:45:12 UTC 
For above #comment31, it is ovs kernel qinq case job. It also has this issue.
Comment 33 liting 2024-01-05 10:28:34 UTC 
For ovs kernel qinq case, following is the detail results.
>>>>>>>>>>>>>>>>i40e driver<<<<<<<<<<<<<<<<<<<<<<
test env: 730-52 25g i40e <--connect--> 730-53 25g i40e
SW Inner kernel VLAN:  ping ipv4 address successfully, but ping ipv6 address not always successfully.
I have added a comment to the following bug.
https://bugzilla.redhat.com/show_bug.cgi?id=1921462

rhel9.2 ovs3.1
https://beaker.engineering.redhat.com/jobs/8752659
rhel9.2 ovs2.17
https://beaker.engineering.redhat.com/jobs/8752408
rhel8.6 ovs2.17
https://beaker.engineering.redhat.com/jobs/8752859
    
test env: 730-52 25g i40e <--connect--> 730-53 25g i40e     
HW outer SR-IOV VF: not tested, it blocked by following bug.
https://bugzilla.redhat.com/show_bug.cgi?id=2229854

750-37 10g ixgbe <--connect--> 10g i40e
SW Inner kernel VLAN: ping both ipv4 and ipv6 address successfully
job: rhel9.2 ovs3.1
https://beaker.engineering.redhat.com/jobs/8751898

>>>>>>>>>>>>ice driver<<<<<<<<<<<<<<<<
test env: netqe45 25g ice <--connect--> netqe46 25g ice
SW Inner kernel VLAN:  ping ipv4 address successfully, but ping ipv6 address not always successfully.
I have added a comment to the following bug.
https://bugzilla.redhat.com/show_bug.cgi?id=1921462

rhel9.2 ovs3.1
https://beaker.engineering.redhat.com/jobs/8752597
rhel9.2 ovs3.2
https://beaker.engineering.redhat.com/jobs/8753510
https://beaker.engineering.redhat.com/jobs/8753511
rhel9.2 ovs2.17
https://beaker.engineering.redhat.com/jobs/8753517
rhel8.6 ovs2.17
https://beaker.engineering.redhat.com/jobs/8756680

test env: netqe45 25g ice <--connect--> netqe46 25g ice
HW outer SR-IOV VF: ping ipv4 address successfully, but ping ipv6 address not always successfully.
I have added a comment to the following bug.
https://bugzilla.redhat.com/show_bug.cgi?id=1921462
rhel9.2 ovs2.17
https://beaker.engineering.redhat.com/jobs/8756248
rhel9.2 ovs3.2
https://beaker.engineering.redhat.com/jobs/8756354
rhel9.2 ovs3.1
https://beaker.engineering.redhat.com/jobs/8756420
rhel8.6 ovs2.17
https://beaker.engineering.redhat.com/jobs/8756638

test env: wsfd-advnetlab10 ice <--connect--> wsfd-advnetlab11 ice
SW Inner kernel VLAN:  ping ipv4 address successfully, but ping ipv6 address not always successfully.
rhel9.2 ovs3.1
https://beaker.engineering.redhat.com/jobs/8756294
rhel9.2 ovs2.17
https://beaker.engineering.redhat.com/jobs/8756169
rhel9.2 ovs3.2
https://beaker.engineering.redhat.com/jobs/8756310
rhel8.6 ovs2.17
https://beaker.engineering.redhat.com/jobs/8756315

test env: wsfd-advnetlab10 ice <--connect--> wsfd-advnetlab11 ice
HW outer SR-IOV VF: ping ipv4 address successfully, but ping ipv6 address not always successfully.
rhel9.2 ovs2.17
https://beaker.engineering.redhat.com/jobs/8755945
rhel9.2 ovs3.1
https://beaker.engineering.redhat.com/jobs/8756670
rhel9.2 ovs3.2
https://beaker.engineering.redhat.com/jobs/8756678
rhel8.6 ovs2.17
https://beaker.engineering.redhat.com/jobs/8756434