Bug 1918309
| Summary: | [failover vf migration][win10 vm] After migrating the source guest, the target guest could not ping the source host successfully(only qemu hits the issue) | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Yanghang Liu <yanghliu> |
| Component: | qemu-kvm | Assignee: | ybendito |
| qemu-kvm sub component: | Networking | QA Contact: | Yanhui Ma <yama> |
| Status: | CLOSED WONTFIX | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | chayang, jinzhao, juzhang, lvivier, mdean, virt-maint, yalzhang, ybendito, yvugenfi |
| Version: | 9.0 | Keywords: | Reopened, Triaged |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Windows | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-06-27 08:09:09 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
After the migration is finished, please exit source qemu, then change the MAC address of the source VF to something different. Does this make a change? One more question: whether the behavior is different with Linux VM? (i.e. after migration you can ping the src host) Meirav - not sure if this ends up as a windows guest or more of a networking issue. Figured I'd sent your way first since Yuri answered already (In reply to ybendito from comment #1) Hi Yuri, > After the migration is finished, please exit source qemu, then change the > MAC address of the source VF to something different. > Does this make a change? Yes. After I change the MAC address of the source VF from 22:2b:62:bb:a9:82 to 22:2b:62:bb:a9:83, the target vm can ping source host succssfully now. (1) on the source host: # ip link set enp6s0f0 vf 0 mac 22:2b:62:bb:a9:83 (2) in target Win10 vm: C:\Windows\system32>ping 10.73.33.242 Pinging 10.73.33.242 with 32 bytes of data: Reply from 10.73.33.242: bytes=32 time<1ms TTL=64 Reply from 10.73.33.242: bytes=32 time<1ms TTL=64 Reply from 10.73.33.242: bytes=32 time<1ms TTL=64 Reply from 10.73.33.242: bytes=32 time<1ms TTL=64 Ping statistics for 10.73.33.242: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms If I change the MAC address back to the original MAC address 22:2b:62:bb:a9:82 , the target vm cannot ping the source host again. (1) on source host: # ip link set enp6s0f0 vf 0 mac 22:2b:62:bb:a9:82 (2) in target win10 vm: C:\Windows\system32>ping 10.73.33.242 Pinging 10.73.33.242 with 32 bytes of data: Reply from 10.73.33.171: Destination host unreachable. Reply from 10.73.33.171: Destination host unreachable. Reply from 10.73.33.171: Destination host unreachable. Reply from 10.73.33.171: Destination host unreachable. Ping statistics for 10.73.33.242: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), > One more question: whether the behavior is different with Linux VM? (i.e. after migration you can ping the src host) After migration , the target linux vm can still ping the source host successfully.(no other additional operations are required) Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release. Removed the ITR from all bugs as part of the change. (In reply to Yanghang Liu from comment #4) > (In reply to ybendito from comment #1) > Hi Yuri, > > > After the migration is finished, please exit source qemu, then change the > > MAC address of the source VF to something different. > > Does this make a change? > > > Yes. > > After I change the MAC address of the source VF from 22:2b:62:bb:a9:82 to > 22:2b:62:bb:a9:83, the target vm can ping source host succssfully now. > > (1) on the source host: > # ip link set enp6s0f0 vf 0 mac 22:2b:62:bb:a9:83 > > (2) in target Win10 vm: > C:\Windows\system32>ping 10.73.33.242 > > Pinging 10.73.33.242 with 32 bytes of data: > Reply from 10.73.33.242: bytes=32 time<1ms TTL=64 > Reply from 10.73.33.242: bytes=32 time<1ms TTL=64 > Reply from 10.73.33.242: bytes=32 time<1ms TTL=64 > Reply from 10.73.33.242: bytes=32 time<1ms TTL=64 > > Ping statistics for 10.73.33.242: > Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), > Approximate round trip times in milli-seconds: > Minimum = 0ms, Maximum = 0ms, Average = 0ms > > > > If I change the MAC address back to the original MAC address > 22:2b:62:bb:a9:82 , the target vm cannot ping the source host again. > > (1) on source host: > # ip link set enp6s0f0 vf 0 mac 22:2b:62:bb:a9:82 > > (2) in target win10 vm: > > C:\Windows\system32>ping 10.73.33.242 > > Pinging 10.73.33.242 with 32 bytes of data: > Reply from 10.73.33.171: Destination host unreachable. > Reply from 10.73.33.171: Destination host unreachable. > Reply from 10.73.33.171: Destination host unreachable. > Reply from 10.73.33.171: Destination host unreachable. > > Ping statistics for 10.73.33.242: > Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), > > > > One more question: whether the behavior is different with Linux VM? (i.e. after migration you can ping the src host) > > After migration , the target linux vm can still ping the source host > successfully.(no other additional operations are required) Hi Yuri, Update results for rhel9 guest with rhel9 qemu and kernel, after migration, linux guest can't ping the source host. After changing vf mac of source host to different one, can ping the source host. (In reply to Yanhui Ma from comment #6) > > > > > One more question: whether the behavior is different with Linux VM? (i.e. after migration you can ping the src host) > > > > After migration , the target linux vm can still ping the source host > > successfully.(no other additional operations are required) > > Hi Yuri, > > Update results for rhel9 guest with rhel9 qemu and kernel, after migration, > linux guest can't ping the source host. After changing vf mac of source host > to different one, can ping the source host. Hi Yuri, If in rhel9 windows's behaviour is same as Linux VM, both can't ping the source host after migration if the vf mac of source host is same as guest. After changing vf mac of source host to different one, can ping the source host. Do you think it is still a bug? And the issue can't be hit by libvirt. After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. Hi Yuri, It seems we need to create new bug in RHEL project of jira if we want to close it migrated. RHELPLAN-64579 is the original bug in RHEL Planning of jira. But based on comment 7, since libvirt can't hit the issue, and after changing src vf mac, also no issue, QE think it can be closed wontfix. How do you think? (In reply to Yanhui Ma from comment #10) > Hi Yuri, > > It seems we need to create new bug in RHEL project of jira if we want to > close it migrated. > RHELPLAN-64579 is the original bug in RHEL Planning of jira. > > But based on comment 7, since libvirt can't hit the issue, and after > changing src vf mac, also no issue, QE think it can be closed wontfix. How > do you think? I agree. The failure to ping definitely related to the fact that the source host's PF already knows this MAC address as internal. Libvirt changes the source VF's MAC, so the problem does not happen. |
Description of problem: After migrating the source Win10 guest with 82599ES, the target Win10 guest with BCM57810 could not ping the source host successfully. Version-Release number of selected component (if applicable): host: qemu-kvm-5.2.0-3.module+el8.4.0+9499+42e58f08.x86_64 4.18.0-275.el8.dt5.x86_64 guest: en_windows_10_business_editions_version_2004_updated_may_2020_x64_dvd_aa8db2cc.iso virtio-win-prewhql-0.1-193 How reproducible: 100% Steps to Reproduce: 1. Create a bridge on source/target host based on the PF source host: # lshw -c network -businfo pci@0000:06:00.0 enp6s0f0 network 82599ES 10-Gigabit SFI/SFP+ Network Connection pci@0000:06:00.1 enp6s0f1 network 82599ES 10-Gigabit SFI/SFP+ Network Connection # nmcli connection add type bridge ifname switch con-name switch stp no # nmcli connection modify "enp130s0f0" master switch # nmcli connection up "enp130s0f0" # ifconfig ... switch: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.73.33.233 netmask 255.255.254.0 broadcast 10.73.33.255 inet6 2620:52:0:4920:9690:30ee:ba96:2fbb prefixlen 64 scopeid 0x0<global> inet6 fe80::7943:a163:c621:1e8d prefixlen 64 scopeid 0x20<link> ether 00:1b:21:c3:d0:3c txqueuelen 1000 (Ethernet) RX packets 432 bytes 35117 (34.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 358 bytes 31526 (30.7 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 target host: # lshw -c network -businfo pci@0000:82:00.0 enp130s0f0 network NetXtreme II BCM57810 10 Gigabit Ethernet pci@0000:82:00.1 enp130s0f1 network NetXtreme II BCM57810 10 Gigabit Ethernet # nmcli connection add type bridge ifname switch con-name switch stp no # nmcli connection modify "enp130s0f0" master switch # nmcli connection up "enp130s0f0" # ifconfig ... switch: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.73.33.244 netmask 255.255.254.0 broadcast 10.73.33.255 inet6 2620:52:0:4920:e6af:cfb:65fe:d17d prefixlen 64 scopeid 0x0<global> inet6 fe80::7f14:bf7a:817f:3b44 prefixlen 64 scopeid 0x20<link> ether 00:0a:f7:05:82:c0 txqueuelen 1000 (Ethernet) RX packets 35 bytes 2841 (2.7 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 50 bytes 7655 (7.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 2. Create a VF and setup the mac address of the VF source host: # echo 1 > /sys/bus/pci/devices/0000\:06\:00.0/sriov_numvfs # echo 0000:06:10.0 > /sys/bus/pci/devices/0000\:06\:10.0/driver/unbind # echo "8086 10ed" > /sys/bus/pci/drivers/vfio-pci/new_id # echo "8086 10ed" > /sys/bus/pci/drivers/vfio-pci/remove_id # ip link set enp6s0f0 vf 0 mac 22:2b:62:bb:a9:82 target host: # echo 1 > /sys/bus/pci/devices/0000\:82\:00.0/sriov_numvfs # echo 0000:82:01.0 > /sys/bus/pci/devices/0000\:82\:01.0/driver/unbind # echo "14e4 16af" > /sys/bus/pci/drivers/vfio-pci/new_id # echo "14e4 16af" > /sys/bus/pci/drivers/vfio-pci/remove_id # ip link set enp130s0f0 vf 0 mac 22:2b:62:bb:a9:82 3. start a src Win10 vm with a failover vf and a failover virtio network device source host: /usr/libexec/qemu-kvm -name Win10 \ -M q35 \ -m 4G \ -nodefaults \ -cpu Haswell-noTSX \ -smp 4 \ -device pcie-root-port,id=root.1,chassis=1,addr=0x2.0,multifunction=on \ -device pcie-root-port,id=root.2,chassis=2,addr=0x2.1 \ -device pcie-root-port,id=root.3,chassis=3,addr=0x2.2 \ -device pcie-root-port,id=root.4,chassis=4,addr=0x2.3 \ -device pcie-root-port,id=root.5,chassis=5,addr=0x2.4 \ -device pcie-root-port,id=root.6,chassis=6,addr=0x2.5 \ -device pcie-root-port,id=root.7,chassis=7,addr=0x2.6 \ -device pcie-root-port,id=root.8,chassis=8,addr=0x2.7 \ -blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/nfs/migration_test/win10.qcow2,node-name=my_file \ -blockdev driver=qcow2,node-name=my,file=my_file \ -device virtio-blk-pci,drive=my,id=virtio-blk0,bus=root.1 \ -vnc :0 \ -vga qxl \ -monitor stdio \ -usb -device usb-tablet \ -boot menu=on \ -qmp tcp:0:5555,server,nowait \ -netdev tap,id=hostnet0,vhost=on \ -device virtio-net-pci,netdev=hostnet0,id=net0,mac=22:2b:62:bb:a9:82,bus=root.3,failover=on \ -device vfio-pci,host=0000:06:10.0,id=hostdev0,bus=root.4,failover_pair_id=net0 \ 4.Download the virtio-win-prewhql package in the Win10 vm and install the VIOPROT protocol # cd virtio-win-prewhql-0.1.193\Win10\amd64 # netcfg -v -l vioprot.inf -c p -i VIOPROT 5.Check the network status in the src Win10 vm 5.1 check the ip address of the src Win10 vm after installing the VIOPROT protocol # ipconfig /all Ethernet adapter Ethernet 2: Connection-specific DNS Suffix . : lab.eng.pek2.redhat.com Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #2 Physical Address. . . . . . . . . : 22-2B-62-BB-A9-82 DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes IPv6 Address. . . . . . . . . . . : 2620:52:0:4920:ddd5:6e85:96a2:41e7(Preferred) Temporary IPv6 Address. . . . . . : 2620:52:0:4920:e58a:2eca:b4c:58c0(Preferred) Link-local IPv6 Address . . . . . : fe80::ddd5:6e85:96a2:41e7%6(Preferred) IPv4 Address. . . . . . . . . . . : 10.73.33.171(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.254.0 Lease Obtained. . . . . . . . . . : Wednesday, January 20, 2021 10:16:45 AM Lease Expires . . . . . . . . . . : Wednesday, January 20, 2021 10:16:45 PM Default Gateway . . . . . . . . . : fe80::46f4:77ff:feb7:6bc1%6 10.73.33.254 DHCP Server . . . . . . . . . . . : 10.73.2.108 DHCPv6 IAID . . . . . . . . . . . : 119679842 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-27-95-FB-F3-9A-7F-BD-26-D1-60 DNS Servers . . . . . . . . . . . : 10.73.2.107 10.73.2.108 10.66.127.10 NetBIOS over Tcpip. . . . . . . . : Enabled 5.2 ping target host successfully # ping 10.73.33.244 Pinging 10.73.33.244 with 32 bytes of data: Reply from 10.73.33.244: bytes=32 time<1ms TTL=64 Reply from 10.73.33.244: bytes=32 time<1ms TTL=64 Reply from 10.73.33.244: bytes=32 time<1ms TTL=64 Reply from 10.73.33.244: bytes=32 time<1ms TTL=64 Ping statistics for 10.73.33.244: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms 5.3 ping source host successfully # ping 10.73.33.233 Pinging 10.73.33.233 with 32 bytes of data: Reply from 10.73.33.233: bytes=32 time<1ms TTL=64 Reply from 10.73.33.233: bytes=32 time<1ms TTL=64 Reply from 10.73.33.233: bytes=32 time<1ms TTL=64 Reply from 10.73.33.233: bytes=32 time<1ms TTL=64 Ping statistics for 10.73.33.233: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms 5.4 ping external network # ping redhat.com Pinging redhat.com [10.4.204.55] with 32 bytes of data: Reply from 10.4.204.55: bytes=32 time=257ms TTL=248 Reply from 10.4.204.55: bytes=32 time=257ms TTL=248 Reply from 10.4.204.55: bytes=32 time=256ms TTL=248 Reply from 10.4.204.55: bytes=32 time=256ms TTL=248 Ping statistics for 10.4.204.55: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 256ms, Maximum = 257ms, Average = 256ms 6.start a dst Win10 vm in listening mode ... -netdev tap,id=hostnet0,vhost=on \ -device virtio-net-pci,netdev=hostnet0,id=net0,mac=22:2b:62:bb:a9:82,bus=root.3,failover=on \ -device vfio-pci,host=0000:82:01.0,id=hostdev0,bus=root.4,failover_pair_id=net0 \ -incoming defer \ # telnet 10.73.33.244 5555 ... {"execute": "migrate-incoming","arguments": {"uri": "tcp:[::]:5800"}} (qemu) info migrate globals: store-global-state: on only-migratable: off send-configuration: on send-section-footer: on decompress-error-check: on clear-bitmap-shift: 18 socket address: [ tcp::::5800 ] 7.migrate the src Win10 vm from source host to target host source host: # telnet 10.73.33.233 5555 ... {"execute": "migrate","arguments":{"uri": "tcp:10.73.33.244:5800"}} {"return": {}} {"timestamp": {"seconds": 1611143546, "microseconds": 649533}, "event": "UNPLUG_PRIMARY", "data": {"device-id": "hostdev0"}} {"timestamp": {"seconds": 1611143571, "microseconds": 642232}, "event": "STOP"} target host: # telnet 10.73.33.244 5555 ... {"timestamp": {"seconds": 1607849328, "microseconds": 746610}, "event": "FAILOVER_NEGOTIATED", "data": {"device-id": "net0"}} {"timestamp": {"seconds": 1607849329, "microseconds": 624741}, "event": "RESUME"} 8. Check the network status in the dst Win10 vm # ipconfig /all Ethernet adapter Ethernet 2: Connection-specific DNS Suffix . : lab.eng.pek2.redhat.com Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter #2 Physical Address. . . . . . . . . : 22-2B-62-BB-A9-82 DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes IPv6 Address. . . . . . . . . . . : 2620:52:0:4920:ddd5:6e85:96a2:41e7(Preferred) Temporary IPv6 Address. . . . . . : 2620:52:0:4920:454b:4a00:1868:7058(Preferred) Link-local IPv6 Address . . . . . : fe80::ddd5:6e85:96a2:41e7%6(Preferred) IPv4 Address. . . . . . . . . . . : 10.73.33.171(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.254.0 Lease Obtained. . . . . . . . . . : Wednesday, January 20, 2021 11:46:26 AM Lease Expires . . . . . . . . . . : Wednesday, January 20, 2021 11:46:25 PM Default Gateway . . . . . . . . . : fe80::46f4:77ff:feb7:6bc1%6 10.73.33.254 DHCP Server . . . . . . . . . . . : 10.73.2.108 DHCPv6 IAID . . . . . . . . . . . : 119679842 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-27-95-FB-F3-9A-7F-BD-26-D1-60 DNS Servers . . . . . . . . . . . : 10.73.2.107 10.73.2.108 10.66.127.10 NetBIOS over Tcpip. . . . . . . . : Enabled 9. do the ping test in the dst Win10 vm: 9.1 ping the target host successfully # ping 10.73.33.244 Pinging 10.73.33.244 with 32 bytes of data: Reply from 10.73.33.244: bytes=32 time=2ms TTL=64 Reply from 10.73.33.244: bytes=32 time<1ms TTL=64 Reply from 10.73.33.244: bytes=32 time<1ms TTL=64 Reply from 10.73.33.244: bytes=32 time<1ms TTL=64 Ping statistics for 10.73.33.244: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 2ms, Average = 0ms 9.2 ping the external network successfully # ping redhat.com Pinging redhat.com [10.4.204.55] with 32 bytes of data: Reply from 10.4.204.55: bytes=32 time=257ms TTL=248 Reply from 10.4.204.55: bytes=32 time=257ms TTL=248 Reply from 10.4.204.55: bytes=32 time=257ms TTL=248 Reply from 10.4.204.55: bytes=32 time=256ms TTL=248 Ping statistics for 10.4.204.55: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 256ms, Maximum = 257ms, Average = 256ms 9.3 fail to ping the source host # ping 10.73.33.233 Pinging 10.73.33.233 with 32 bytes of data: Reply from 10.73.33.171: Destination host unreachable. Reply from 10.73.33.171: Destination host unreachable. Reply from 10.73.33.171: Destination host unreachable. Reply from 10.73.33.171: Destination host unreachable. Ping statistics for 10.73.33.233: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Actual results: After migrating the source guest with 82599ES, the target guest could not ping the source host successfully Expected results: the target guest could ping the source host successfully Additional info: (1)Download the driver of 82599ES in Win10 vm from https://downloadcenter.intel.com/download/22283 (2)The target host can always ping the source host successfully