Bug 1969306

Summary: [failover vf][RHEL85_vm] the failover vf and failover virtio nic with the same MAC address have different valid IP addresses
Product: Red Hat Enterprise Linux 9 Reporter: Yanghang Liu <yanghliu>
Component: NetworkManagerAssignee: NetworkManager Development Team <nm-team>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Desktop QE <desktop-qa-list>
Severity: medium Docs Contact:
Priority: medium    
Version: 9.0CC: aadam, bgalvani, chayang, ferferna, jinzhao, juzhang, lrintel, lvivier, rkhan, sfaye, sukulkar, till, virt-maint, wquan, yalzhang, yama
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-24 11:54:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
the current-configuration of the private swicth which PF is connected to
none
vm kernel dmesg log none

Description Yanghang Liu 2021-06-08 07:36:37 UTC 
Description of problem:
In the RHEL vm, the failover vf and failover virtio nic with the same MAC address may have different valid IP addresses.

Version-Release number of selected component (if applicable):
host:
qemu-kvm-6.0.0-18.module+el8.5.0+11243+5269aaa1.x86_64
libvirt-7.3.0-1.module+el8.5.0+11004+f4810536.x86_64
4.18.0-310.el8.x86_64

guest:
4.18.0-310.el8.x86_64

How reproducible:
> 50% 

Steps to Reproduce:
1. create bridge named br0 based on the PF

nmcli connection add type bridge ifname br0 con-name br0 stp off autoconnect yes
nmcli connection add type bridge-slave ifname "$MAIN_CONN" con-name "$MAIN_CONN" master br0 autoconnect yes
systemctl restart NetworkManager


# brctl show 
bridge name	bridge id		STP enabled	interfaces
br0		8000.3cfdfe1590aa	no		enp6s0f1


2. setup vm network 

# virsh net-dumpxml failover-bridge 
<network>
  <name>failover-bridge</name>
  <uuid>ddc85097-aad6-4f2c-b2d7-83f5bc88c884</uuid>
  <forward mode='bridge'/>
  <bridge name='br0'/>
</network>

# virsh net-dumpxml failover-vf 
<network>
  <name>failover-vf</name>
  <uuid>8e0514d1-dc3b-4f4e-8f4e-b50040fe5197</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x06' slot='0x0a' function='0x0'/>
  </forward>
</network>

# virsh net-dumpxml failover-vf  --inactive 
<network>
  <name>failover-vf</name>
  <uuid>46e42769-1513-4967-8889-c6f5216ba212</uuid>
  <forward mode='hostdev' managed='yes'>
    <pf dev='enp6s0f1'/>
  </forward>
</network>



3. start a VM with a failover vf and a virtio net device

    <interface type='network'>
      <mac address='52:54:11:aa:1c:ef'/>
      <source network='failover-bridge'/>
      <model type='virtio'/>
      <teaming type='persistent'/>
      <alias name='ua-test'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>

    <interface type='network'>
      <mac address='52:54:11:aa:1c:ef'/>
      <source network='failover-vf'/>
      <teaming type='transient' persistent='ua-test'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </interface>


the related qemu-kcm cmd line:
-device virtio-net-pci,failover=on,netdev=hostua-test,id=ua-test,mac=52:54:11:aa:1c:ef,bus=pci.4,addr=0x0 \
-device vfio-pci,host=0000:06:0a.0,id=hostdev0,bus=pci.5,addr=0x0,failover_pair_id=ua-test \


4. check the ip address of the network interface in the vm
# ifconfig 
...
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.200.66  netmask 255.255.255.0  broadcast 192.168.200.255
        inet6 fe80::227a:3e6f:9f6:f793  prefixlen 64  scopeid 0x20<link>
        inet6 2001::2d80:cc27:b222:d3e8  prefixlen 64  scopeid 0x0<global>
        ether 52:54:11:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 158  bytes 18336 (17.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 73  bytes 10430 (10.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp4s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::155d:289a:159e:a2f3  prefixlen 64  scopeid 0x20<link>
        ether 52:54:11:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 122  bytes 12036 (11.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 31  bytes 6278 (6.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.200.73  netmask 255.255.255.0  broadcast 192.168.200.255
        inet6 fe80::7a86:7841:3c6c:4872  prefixlen 64  scopeid 0x20<link>
        ether 52:54:11:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 36  bytes 6300 (6.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 42  bytes 4152 (4.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



Actual results:
The failover vf and failover virtio nic with the same MAC address *may have different valid IP addresses.*


Expected results:
The failover vf and failover virtio nic with the same MAC address *should always have the same valid IP addresses.*

Additional info:
(1) The same problem that was found on the windows vm has been fixed

Bug 1918594 - [failover vf] In the vm, the failover vf and failover virtio nic with the same MAC address have different valid IP addresses

(2) This bug can be reproduced by just starting/restarting the RHEL vm multiple times

(3) The PF is connected to a private switch that I have configured with DHCP and VLAN services.

The switch configure details is in the attachment.
Comment 1 Yanghang Liu 2021-06-08 07:50:46 UTC 
Created attachment 1789336 [details]
the current-configuration of the private swicth which PF is connected to
Comment 2 Laurent Vivier 2021-06-14 19:21:55 UTC 
Yanghang,

Could you provide the logs of the VM kernel?
Could you check the DHCP server on the bridge network is the same server as on the PF network?
Could you provide logs of the DHCP server to see why it doesn't provide the same IP address for the same MAC address?

Thanks
Comment 3 Yanghang Liu 2021-06-16 10:03:42 UTC 
Created attachment 1791499 [details]
vm kernel dmesg log
Comment 4 Yanghang Liu 2021-06-16 11:12:23 UTC 
(In reply to Laurent Vivier from comment #2)
> Yanghang,

Hi Laurent,


> Could you provide the logs of the VM kernel?

The vm dmesg is in the attachment.
 
> Could you check the DHCP server on the bridge network is the same server as on the PF network?

As the following information indicates， the failover vf network and the bridge network are both created based on the same PF(enp6s0f1).

# brctl show 
bridge name	bridge id		STP enabled	interfaces
br0		8000.3cfdfe1590aa	no		enp6s0f1


# virsh net-dumpxml failover-bridge 
<network>
  <name>failover-bridge</name>
  <uuid>ddc85097-aad6-4f2c-b2d7-83f5bc88c884</uuid>
  <forward mode='bridge'/>
  <bridge name='br0'/>
</network>

# virsh net-dumpxml failover-vf  --inactive 
<network>
  <name>failover-vf</name>
  <uuid>46e42769-1513-4967-8889-c6f5216ba212</uuid>
  <forward mode='hostdev' managed='yes'>
    <pf dev='enp6s0f1'/>
  </forward>
</network>


The PF(enp6s0f1) is connected to a private switch that configured with DHCP service.
 

> Could you provide logs of the DHCP server to see why it doesn't provide the same IP address for the same MAC address?


I configured the DHCP service on a private switch(the detailed configuration of this switch is in the attachment)

If possible , could you please tell me which log "DHCP server log" refers to ?
Comment 5 Yanghang Liu 2021-06-16 11:24:41 UTC 
According to the infor I got from the switch's arp table, *the failover virtio net devcie* and *the failover vf* with the same MAC address are indeed assigned different IP addresses:

（The reason for the difference between the IP address here and the one in the bug description is because I restarted the vm , but the same problem can still be reproduced.）


<H3C>display arp
  Type: S-Static   D-Dynamic   O-Openflow   R-Rule   M-Multiport  I-Invalid
IP address      MAC address    VLAN     Interface                Aging Type 
192.168.200.10  3cfd-fe15-90aa 200      FGE1/0/42                5     D     <-- the br0 on the host
192.168.200.115 5254-11aa-1cef 200      FGE1/0/42                20    D     <-- the failvoer virtio net devcie in the vm
192.168.200.197 5254-11aa-1cef 200      FGE1/0/42                20    D     <-- the failover vf in the vm
Comment 6 Laurent Vivier 2021-06-17 14:06:37 UTC 
(In reply to Yanghang Liu from comment #4)
....
> If possible , could you please tell me which log "DHCP server log" refers to
> ?

What I try to understand is:

1- why enp5s0 is asking for an IP address?

   in the VM:

   Could you check if /etc/sysconfig/network-scripts/ifcfg-enp5s0 exists? (it should not)?
   There should not be any connection configured to enp5s0 and "nmcli con show enp5s0" should fail
   Could you check the log of NetworkManager?
   (something like "journalctl -b 0 -t NetworkManager -o short", you can add "| grep dhcp" at the end to check only DHCP)

2- why the IP address provided is not the same one?

   Normally the same MAC address implies the same IP address, so I guess there is two DHCP servers, one local on the host bound to the bridge br0 and one bound to the physical network.

   Could you check if dnsmasq is running on the host and if it is providing the IP address?

   journalctl -b 0 -t dnsmasq-dhcp -o short|grep 52:54:11:aa:1c:ef
  
   Ideally you should also check the logs of your private switch bound to PF(enp6s0f1) to see if it provides the IP address too.

To fix "1-", you can do "nmcli con del enp5s0"

To Fix "2", you must find the dnsmasq.conf file that is used (by default "/etc/dnsmasq.conf", but check "ps -ef|grep dnsmasq" to see if there is a "--conf-file" parameter). In the configuration file, check "interface=" is NOT "br0" (the name of you bridge in comment #0)

Thanks
Comment 7 Yanghang Liu 2021-06-25 10:06:04 UTC 
Update the "Steps to Reproduce":

(1) create bridge named br0 based on the PF

(2) setup vm network

(3) start a vm with only with a failover virtio net device

(4) hot-plug a failover vf into the vm
Comment 8 Yanghang Liu 2021-06-28 04:05:21 UTC 
(In reply to Laurent Vivier from comment #6)

Hi Laurent，

Thanks for the info.

> What I try to understand is:
> 
> 1- why enp5s0 is asking for an IP address?
> 
>    in the VM:
> 
>    Could you check if /etc/sysconfig/network-scripts/ifcfg-enp5s0 exists? (it should not)?
>    There should not be any connection configured to enp5s0 and "nmcli con show enp5s0" should fail
>    Could you check the log of NetworkManager?
>    (something like "journalctl -b 0 -t NetworkManager -o short", you can add "| grep dhcp" at the end to check only DHCP)

>  To fix "1-", you can do "nmcli con del enp5s0"

For 1 , I can prevent the "failover vf"(enp5s0) from getting an IP address by deleting the related connection.

And I will do some detailed tests to see if this behavior is working properly in other failover vf migration test scenarios.

> 2- why the IP address provided is not the same one?

For 2, I have not found the root cause of this problem yet.

>    Normally the same MAC address implies the same IP address, so I guess there is two DHCP servers, one local on the host bound to the bridge br0 and one bound to the physical network.
> 
>    Could you check if dnsmasq is running on the host and if it is providing the IP address?

>    journalctl -b 0 -t dnsmasq-dhcp -o short|grep 52:54:11:aa:1c:ef

The dnsmasq on the host is not provided any IP address for 52:54:11:aa:1c:ef.

The related command info output is as following:
# journalctl -b 0 -t dnsmasq-dhcp -o short|grep 52:54:11:aa:1c:ef  <--- There is no output after running this command
   
>    Ideally you should also check the logs of your private switch bound to PF(enp6s0f1) to see if it provides the IP address too.

As I mentioned in comment 5, I can see that the same MAC address has two different IP addresses on the arp table of the private switch 

> To Fix "2", you must find the dnsmasq.conf file that is used (by default "/etc/dnsmasq.conf", but check "ps -ef|grep dnsmasq" to see if there is a "--conf-file" parameter). 
> In the configuration file, check "interface=" is NOT "br0" (the name of you bridge in comment #0)


The related command info output is as following:

# ps -ef|grep dnsmasq
dnsmasq     2401       1  0 18:35 ?        00:00:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
root        2402    2401  0 18:35 ?        00:00:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

# cat /var/lib/libvirt/dnsmasq/default.conf | grep -i interface
except-interface=lo
interface=virbr0  <---- the interface is "virtbr0" instead of "br0"
Comment 9 Yanhui Ma 2021-09-01 09:22:24 UTC 
The issue can be easily reproduced by live migration between rhel8.4.0 and rhel8.5.0, almost 100% reproduce ratio.

For steps and package version, pls refer to https://bugzilla.redhat.com/show_bug.cgi?id=1999927#c0, just change win10 guest to rhel8.4.0 guest.

# ifconfig 
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.74.5  netmask 255.255.252.0  broadcast 10.73.75.255
        inet6 2620:52:0:4948:f68e:38ff:fec3:8080  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::f68e:38ff:fec3:8080  prefixlen 64  scopeid 0x20<link>
        ether f4:8e:38:c3:80:80  txqueuelen 1000  (Ethernet)
        RX packets 7582  bytes 548637 (535.7 KiB)
        RX errors 0  dropped 169  overruns 0  frame 0
        TX packets 151  bytes 15631 (15.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.200.222  netmask 255.255.255.0  broadcast 192.168.200.255
        inet6 fe80::e9e9:9c38:3fb5:bd29  prefixlen 64  scopeid 0x20<link>
        inet6 2001::6398:cd12:a87c:754c  prefixlen 64  scopeid 0x0<global>
        ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
        RX packets 126  bytes 15440 (15.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 44  bytes 7372 (7.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp4s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
        RX packets 198  bytes 23023 (22.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 52  bytes 9280 (9.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.200.223  netmask 255.255.255.0  broadcast 192.168.200.255
        inet6 fe80::9c3c:b999:e0b6:65f3  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
        RX packets 8  bytes 1640 (1.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4  bytes 1112 (1.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
Comment 10 John Ferlan 2021-09-08 21:21:19 UTC 
Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.  Removed the ITR from all bugs as part of the change.
Comment 11 Laurent Vivier 2021-09-13 16:56:05 UTC 
(In reply to Yanhui Ma from comment #9)
> The issue can be easily reproduced by live migration between rhel8.4.0 and
> rhel8.5.0, almost 100% reproduce ratio.
> 
> For steps and package version, pls refer to
> https://bugzilla.redhat.com/show_bug.cgi?id=1999927#c0, just change win10
> guest to rhel8.4.0 guest.
> 
> # ifconfig 
> enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 10.73.74.5  netmask 255.255.252.0  broadcast 10.73.75.255
>         inet6 2620:52:0:4948:f68e:38ff:fec3:8080  prefixlen 64  scopeid
> 0x0<global>
>         inet6 fe80::f68e:38ff:fec3:8080  prefixlen 64  scopeid 0x20<link>
>         ether f4:8e:38:c3:80:80  txqueuelen 1000  (Ethernet)
>         RX packets 7582  bytes 548637 (535.7 KiB)
>         RX errors 0  dropped 169  overruns 0  frame 0
>         TX packets 151  bytes 15631 (15.2 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 192.168.200.222  netmask 255.255.255.0  broadcast
> 192.168.200.255
>         inet6 fe80::e9e9:9c38:3fb5:bd29  prefixlen 64  scopeid 0x20<link>
>         inet6 2001::6398:cd12:a87c:754c  prefixlen 64  scopeid 0x0<global>
>         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
>         RX packets 126  bytes 15440 (15.0 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 44  bytes 7372 (7.1 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> enp4s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
>         RX packets 198  bytes 23023 (22.4 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 52  bytes 9280 (9.0 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 192.168.200.223  netmask 255.255.255.0  broadcast
> 192.168.200.255
>         inet6 fe80::9c3c:b999:e0b6:65f3  prefixlen 64  scopeid 0x20<link>
>         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
>         RX packets 8  bytes 1640 (1.6 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 4  bytes 1112 (1.0 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Could you check you DON'T have the file /etc/sysconfig/network-scripts/ifcfg-enp5s0 ?

Thanks
Comment 12 Yanhui Ma 2021-09-15 14:21:52 UTC 
(In reply to Laurent Vivier from comment #11)
> (In reply to Yanhui Ma from comment #9)
> > The issue can be easily reproduced by live migration between rhel8.4.0 and
> > rhel8.5.0, almost 100% reproduce ratio.
> > 
> > For steps and package version, pls refer to
> > https://bugzilla.redhat.com/show_bug.cgi?id=1999927#c0, just change win10
> > guest to rhel8.4.0 guest.
> > 
> > # ifconfig 
> > enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> >         inet 10.73.74.5  netmask 255.255.252.0  broadcast 10.73.75.255
> >         inet6 2620:52:0:4948:f68e:38ff:fec3:8080  prefixlen 64  scopeid
> > 0x0<global>
> >         inet6 fe80::f68e:38ff:fec3:8080  prefixlen 64  scopeid 0x20<link>
> >         ether f4:8e:38:c3:80:80  txqueuelen 1000  (Ethernet)
> >         RX packets 7582  bytes 548637 (535.7 KiB)
> >         RX errors 0  dropped 169  overruns 0  frame 0
> >         TX packets 151  bytes 15631 (15.2 KiB)
> >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > 
> > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> >         inet 192.168.200.222  netmask 255.255.255.0  broadcast
> > 192.168.200.255
> >         inet6 fe80::e9e9:9c38:3fb5:bd29  prefixlen 64  scopeid 0x20<link>
> >         inet6 2001::6398:cd12:a87c:754c  prefixlen 64  scopeid 0x0<global>
> >         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
> >         RX packets 126  bytes 15440 (15.0 KiB)
> >         RX errors 0  dropped 0  overruns 0  frame 0
> >         TX packets 44  bytes 7372 (7.1 KiB)
> >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > 
> > enp4s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> >         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
> >         RX packets 198  bytes 23023 (22.4 KiB)
> >         RX errors 0  dropped 0  overruns 0  frame 0
> >         TX packets 52  bytes 9280 (9.0 KiB)
> >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > 
> > enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> >         inet 192.168.200.223  netmask 255.255.255.0  broadcast
> > 192.168.200.255
> >         inet6 fe80::9c3c:b999:e0b6:65f3  prefixlen 64  scopeid 0x20<link>
> >         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
> >         RX packets 8  bytes 1640 (1.6 KiB)
> >         RX errors 0  dropped 0  overruns 0  frame 0
> >         TX packets 4  bytes 1112 (1.0 KiB)
> >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> Could you check you DON'T have the file
> /etc/sysconfig/network-scripts/ifcfg-enp5s0 ?
> 

Yes, only ifcfg-enp1s0 file exists.


# cd /etc/sysconfig/network-scripts/
# ls
ifcfg-enp1s0
# cat ifcfg-enp1s0 
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=enp1s0
UUID=d7c9441b-de89-4b4f-84c1-69b88ff06c38
DEVICE=enp1s0
ONBOOT=yes
> Thanks
Comment 13 Laurent Vivier 2021-09-20 13:20:35 UTC 
(In reply to Yanhui Ma from comment #12)
> (In reply to Laurent Vivier from comment #11)
> > (In reply to Yanhui Ma from comment #9)
> > > The issue can be easily reproduced by live migration between rhel8.4.0 and
> > > rhel8.5.0, almost 100% reproduce ratio.
> > > 
> > > For steps and package version, pls refer to
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1999927#c0, just change win10
> > > guest to rhel8.4.0 guest.
> > > 
> > > # ifconfig 
> > > enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> > >         inet 10.73.74.5  netmask 255.255.252.0  broadcast 10.73.75.255
> > >         inet6 2620:52:0:4948:f68e:38ff:fec3:8080  prefixlen 64  scopeid
> > > 0x0<global>
> > >         inet6 fe80::f68e:38ff:fec3:8080  prefixlen 64  scopeid 0x20<link>
> > >         ether f4:8e:38:c3:80:80  txqueuelen 1000  (Ethernet)
> > >         RX packets 7582  bytes 548637 (535.7 KiB)
> > >         RX errors 0  dropped 169  overruns 0  frame 0
> > >         TX packets 151  bytes 15631 (15.2 KiB)
> > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > 
> > > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> > >         inet 192.168.200.222  netmask 255.255.255.0  broadcast
> > > 192.168.200.255
> > >         inet6 fe80::e9e9:9c38:3fb5:bd29  prefixlen 64  scopeid 0x20<link>
> > >         inet6 2001::6398:cd12:a87c:754c  prefixlen 64  scopeid 0x0<global>
> > >         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
> > >         RX packets 126  bytes 15440 (15.0 KiB)
> > >         RX errors 0  dropped 0  overruns 0  frame 0
> > >         TX packets 44  bytes 7372 (7.1 KiB)
> > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > 
> > > enp4s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> > >         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
> > >         RX packets 198  bytes 23023 (22.4 KiB)
> > >         RX errors 0  dropped 0  overruns 0  frame 0
> > >         TX packets 52  bytes 9280 (9.0 KiB)
> > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > 
> > > enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> > >         inet 192.168.200.223  netmask 255.255.255.0  broadcast
> > > 192.168.200.255
> > >         inet6 fe80::9c3c:b999:e0b6:65f3  prefixlen 64  scopeid 0x20<link>
> > >         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
> > >         RX packets 8  bytes 1640 (1.6 KiB)
> > >         RX errors 0  dropped 0  overruns 0  frame 0
> > >         TX packets 4  bytes 1112 (1.0 KiB)
> > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > 
> > Could you check you DON'T have the file
> > /etc/sysconfig/network-scripts/ifcfg-enp5s0 ?
> > 
> 
> Yes, only ifcfg-enp1s0 file exists.
> 
> 
> # cd /etc/sysconfig/network-scripts/
> # ls
> ifcfg-enp1s0
> # cat ifcfg-enp1s0 
> TYPE=Ethernet
> PROXY_METHOD=none
> BROWSER_ONLY=no
> BOOTPROTO=dhcp
> DEFROUTE=yes
> IPV4_FAILURE_FATAL=no
> IPV6INIT=yes
> IPV6_AUTOCONF=yes
> IPV6_DEFROUTE=yes
> IPV6_FAILURE_FATAL=no
> NAME=enp1s0
> UUID=d7c9441b-de89-4b4f-84c1-69b88ff06c38
> DEVICE=enp1s0
> ONBOOT=yes
> > Thanks


This is really weird. Who configure enp4s0 and enp5s0 then?

Do you have results for "nmcli con show enp4s0 enp5s0"?
Comment 14 Yanhui Ma 2021-09-22 09:03:33 UTC 
(In reply to Laurent Vivier from comment #13)
> (In reply to Yanhui Ma from comment #12)
> > (In reply to Laurent Vivier from comment #11)
> > > (In reply to Yanhui Ma from comment #9)
> > > > The issue can be easily reproduced by live migration between rhel8.4.0 and
> > > > rhel8.5.0, almost 100% reproduce ratio.
> > > > 
> > > > For steps and package version, pls refer to
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=1999927#c0, just change win10
> > > > guest to rhel8.4.0 guest.
> > > > 
> > > > # ifconfig 
> > > > enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> > > >         inet 10.73.74.5  netmask 255.255.252.0  broadcast 10.73.75.255
> > > >         inet6 2620:52:0:4948:f68e:38ff:fec3:8080  prefixlen 64  scopeid
> > > > 0x0<global>
> > > >         inet6 fe80::f68e:38ff:fec3:8080  prefixlen 64  scopeid 0x20<link>
> > > >         ether f4:8e:38:c3:80:80  txqueuelen 1000  (Ethernet)
> > > >         RX packets 7582  bytes 548637 (535.7 KiB)
> > > >         RX errors 0  dropped 169  overruns 0  frame 0
> > > >         TX packets 151  bytes 15631 (15.2 KiB)
> > > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > > 
> > > > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> > > >         inet 192.168.200.222  netmask 255.255.255.0  broadcast
> > > > 192.168.200.255
> > > >         inet6 fe80::e9e9:9c38:3fb5:bd29  prefixlen 64  scopeid 0x20<link>
> > > >         inet6 2001::6398:cd12:a87c:754c  prefixlen 64  scopeid 0x0<global>
> > > >         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
> > > >         RX packets 126  bytes 15440 (15.0 KiB)
> > > >         RX errors 0  dropped 0  overruns 0  frame 0
> > > >         TX packets 44  bytes 7372 (7.1 KiB)
> > > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > > 
> > > > enp4s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> > > >         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
> > > >         RX packets 198  bytes 23023 (22.4 KiB)
> > > >         RX errors 0  dropped 0  overruns 0  frame 0
> > > >         TX packets 52  bytes 9280 (9.0 KiB)
> > > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > > 
> > > > enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> > > >         inet 192.168.200.223  netmask 255.255.255.0  broadcast
> > > > 192.168.200.255
> > > >         inet6 fe80::9c3c:b999:e0b6:65f3  prefixlen 64  scopeid 0x20<link>
> > > >         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
> > > >         RX packets 8  bytes 1640 (1.6 KiB)
> > > >         RX errors 0  dropped 0  overruns 0  frame 0
> > > >         TX packets 4  bytes 1112 (1.0 KiB)
> > > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > 
> > > Could you check you DON'T have the file
> > > /etc/sysconfig/network-scripts/ifcfg-enp5s0 ?
> > > 
> > 
> > Yes, only ifcfg-enp1s0 file exists.
> > 
> > 
> > # cd /etc/sysconfig/network-scripts/
> > # ls
> > ifcfg-enp1s0
> > # cat ifcfg-enp1s0 
> > TYPE=Ethernet
> > PROXY_METHOD=none
> > BROWSER_ONLY=no
> > BOOTPROTO=dhcp
> > DEFROUTE=yes
> > IPV4_FAILURE_FATAL=no
> > IPV6INIT=yes
> > IPV6_AUTOCONF=yes
> > IPV6_DEFROUTE=yes
> > IPV6_FAILURE_FATAL=no
> > NAME=enp1s0
> > UUID=d7c9441b-de89-4b4f-84c1-69b88ff06c38
> > DEVICE=enp1s0
> > ONBOOT=yes
> > > Thanks
> 
> 
> This is really weird. Who configure enp4s0 and enp5s0 then?
> 

Hi Laurent,

Please note the additional info in comment 0, my environment is the same with that.
(3) The PF is connected to a private switch that I have configured with DHCP and VLAN services.
The switch configure details is in the attachment.



> Do you have results for "nmcli con show enp4s0 enp5s0"?

# nmcli c show
NAME                UUID                                  TYPE      DEVICE     
Wired connection 2  52fbcc8b-0f0f-3553-a693-aa3ab0052240  ethernet  enp4s0nsby 
enp1s0              d7c9441b-de89-4b4f-84c1-69b88ff06c38  ethernet  enp1s0     
Wired connection 1  b8269223-09ba-3433-b75f-638e3f87d4c7  ethernet  enp4s0     
Wired connection 3  f4b3782e-a253-3d50-828f-9936798f6f41  ethernet  enp5s0

# nmcli c show "Wired connection 1"
connection.id:                          Wired connection 1
connection.uuid:                        b8269223-09ba-3433-b75f-638e3f87d4c7
connection.stable-id:                   --
connection.type:                        802-3-ethernet
connection.interface-name:              enp4s0
connection.autoconnect:                 yes
connection.autoconnect-priority:        -999
connection.autoconnect-retries:         -1 (default)
connection.multi-connect:               0 (default)
connection.auth-retries:                -1
connection.timestamp:                   1632300281
connection.read-only:                   no
connection.permissions:                 --
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.autoconnect-slaves:          -1 (default)
connection.secondaries:                 --
connection.gateway-ping-timeout:        0
connection.metered:                     unknown
connection.lldp:                        default
connection.mdns:                        -1 (default)
connection.llmnr:                       -1 (default)
connection.wait-device-timeout:         -1
802-3-ethernet.port:                    --
802-3-ethernet.speed:                   0
802-3-ethernet.duplex:                  --
802-3-ethernet.auto-negotiate:          no
802-3-ethernet.mac-address:             --
802-3-ethernet.cloned-mac-address:      --
802-3-ethernet.generate-mac-address-mask:--
802-3-ethernet.mac-address-blacklist:   --
802-3-ethernet.mtu:                     auto
802-3-ethernet.s390-subchannels:        --
802-3-ethernet.s390-nettype:            --
802-3-ethernet.s390-options:            --
802-3-ethernet.wake-on-lan:             default
802-3-ethernet.wake-on-lan-password:    --
802-3-ethernet.accept-all-mac-addresses:-1 (default)
ipv4.method:                            auto
ipv4.dns:                               --
ipv4.dns-search:                        --
ipv4.dns-options:                       --
ipv4.dns-priority:                      0
ipv4.addresses:                         --
ipv4.gateway:                           --
ipv4.routes:                            --
ipv4.route-metric:                      -1
ipv4.route-table:                       0 (unspec)
ipv4.routing-rules:                     --
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-iaid:                         --
ipv4.dhcp-timeout:                      0 (default)
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.dhcp-fqdn:                         --
ipv4.dhcp-hostname-flags:               0x0 (none)
ipv4.never-default:                     no
ipv4.may-fail:                          yes
ipv4.dad-timeout:                       -1 (default)
ipv4.dhcp-vendor-class-identifier:      --
ipv4.dhcp-reject-servers:               --
ipv6.method:                            auto
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       --
ipv6.dns-priority:                      0
ipv6.addresses:                         --
ipv6.gateway:                           --
ipv6.routes:                            --
ipv6.route-metric:                      -1
ipv6.route-table:                       0 (unspec)
ipv6.routing-rules:                     --
ipv6.ignore-auto-routes:                no
ipv6.ignore-auto-dns:                   no
ipv6.never-default:                     no
ipv6.may-fail:                          yes
ipv6.ip6-privacy:                       -1 (unknown)
ipv6.addr-gen-mode:                     stable-privacy
ipv6.ra-timeout:                        0 (default)
ipv6.dhcp-duid:                         --
ipv6.dhcp-iaid:                         --
ipv6.dhcp-timeout:                      0 (default)
ipv6.dhcp-send-hostname:                yes
ipv6.dhcp-hostname:                     --
ipv6.dhcp-hostname-flags:               0x0 (none)
ipv6.token:                             --
proxy.method:                           none
proxy.browser-only:                     no
proxy.pac-url:                          --
proxy.pac-script:                       --
GENERAL.NAME:                           Wired connection 1
GENERAL.UUID:                           b8269223-09ba-3433-b75f-638e3f87d4c7
GENERAL.DEVICES:                        enp4s0
GENERAL.IP-IFACE:                       enp4s0
GENERAL.STATE:                          activated
GENERAL.DEFAULT:                        no
GENERAL.DEFAULT6:                       no
GENERAL.SPEC-OBJECT:                    --
GENERAL.VPN:                            no
GENERAL.DBUS-PATH:                      /org/freedesktop/NetworkManager/ActiveConnection/2
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/Settings/2
GENERAL.ZONE:                           --
GENERAL.MASTER-PATH:                    --
IP4.ADDRESS[1]:                         192.168.200.97/24
IP4.GATEWAY:                            192.168.200.254
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.200.254, mt = 101
IP4.ROUTE[2]:                           dst = 192.168.200.0/24, nh = 0.0.0.0, mt = 101
IP4.DNS[1]:                             192.168.200.253
DHCP4.OPTION[1]:                        dhcp_lease_time = 86400
DHCP4.OPTION[2]:                        dhcp_server_identifier = 192.168.200.254
DHCP4.OPTION[3]:                        domain_name_servers = 192.168.200.253
DHCP4.OPTION[4]:                        expiry = 1632385782
DHCP4.OPTION[5]:                        ip_address = 192.168.200.97
DHCP4.OPTION[6]:                        requested_broadcast_address = 1
DHCP4.OPTION[7]:                        requested_domain_name = 1
DHCP4.OPTION[8]:                        requested_domain_name_servers = 1
DHCP4.OPTION[9]:                        requested_domain_search = 1
DHCP4.OPTION[10]:                       requested_host_name = 1
DHCP4.OPTION[11]:                       requested_interface_mtu = 1
DHCP4.OPTION[12]:                       requested_ms_classless_static_routes = 1
DHCP4.OPTION[13]:                       requested_nis_domain = 1
DHCP4.OPTION[14]:                       requested_nis_servers = 1
DHCP4.OPTION[15]:                       requested_ntp_servers = 1
DHCP4.OPTION[16]:                       requested_rfc3442_classless_static_routes = 1
DHCP4.OPTION[17]:                       requested_root_path = 1
DHCP4.OPTION[18]:                       requested_routers = 1
DHCP4.OPTION[19]:                       requested_static_routes = 1
DHCP4.OPTION[20]:                       requested_subnet_mask = 1
DHCP4.OPTION[21]:                       requested_time_offset = 1
DHCP4.OPTION[22]:                       requested_wpad = 1
DHCP4.OPTION[23]:                       routers = 192.168.200.254
DHCP4.OPTION[24]:                       subnet_mask = 255.255.255.0
IP6.ADDRESS[1]:                         2001::9c00:23d2:ae03:2efb/64
IP6.ADDRESS[2]:                         fe80::ef5f:2277:95dd:73ad/64
IP6.GATEWAY:                            fe80::b2f9:63ff:feb8:6d37
IP6.ROUTE[1]:                           dst = 2001::/64, nh = ::, mt = 101
IP6.ROUTE[2]:                           dst = ::/0, nh = fe80::b2f9:63ff:feb8:6d37, mt = 101
IP6.ROUTE[3]:                           dst = fe80::/64, nh = ::, mt = 101


# nmcli c show "Wired connection 3"
connection.id:                          Wired connection 3
connection.uuid:                        f4b3782e-a253-3d50-828f-9936798f6f41
connection.stable-id:                   --
connection.type:                        802-3-ethernet
connection.interface-name:              enp5s0
connection.autoconnect:                 yes
connection.autoconnect-priority:        -999
connection.autoconnect-retries:         -1 (default)
connection.multi-connect:               0 (default)
connection.auth-retries:                -1
connection.timestamp:                   1632300281
connection.read-only:                   no
connection.permissions:                 --
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.autoconnect-slaves:          -1 (default)
connection.secondaries:                 --
connection.gateway-ping-timeout:        0
connection.metered:                     unknown
connection.lldp:                        default
connection.mdns:                        -1 (default)
connection.llmnr:                       -1 (default)
connection.wait-device-timeout:         -1
802-3-ethernet.port:                    --
802-3-ethernet.speed:                   0
802-3-ethernet.duplex:                  --
802-3-ethernet.auto-negotiate:          no
802-3-ethernet.mac-address:             --
802-3-ethernet.cloned-mac-address:      --
802-3-ethernet.generate-mac-address-mask:--
802-3-ethernet.mac-address-blacklist:   --
802-3-ethernet.mtu:                     auto
802-3-ethernet.s390-subchannels:        --
802-3-ethernet.s390-nettype:            --
802-3-ethernet.s390-options:            --
802-3-ethernet.wake-on-lan:             default
802-3-ethernet.wake-on-lan-password:    --
802-3-ethernet.accept-all-mac-addresses:-1 (default)
ipv4.method:                            auto
ipv4.dns:                               --
ipv4.dns-search:                        --
ipv4.dns-options:                       --
ipv4.dns-priority:                      0
ipv4.addresses:                         --
ipv4.gateway:                           --
ipv4.routes:                            --
ipv4.route-metric:                      -1
ipv4.route-table:                       0 (unspec)
ipv4.routing-rules:                     --
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-iaid:                         --
ipv4.dhcp-timeout:                      0 (default)
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.dhcp-fqdn:                         --
ipv4.dhcp-hostname-flags:               0x0 (none)
ipv4.never-default:                     no
ipv4.may-fail:                          yes
ipv4.dad-timeout:                       -1 (default)
ipv4.dhcp-vendor-class-identifier:      --
ipv4.dhcp-reject-servers:               --
ipv6.method:                            auto
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       --
ipv6.dns-priority:                      0
ipv6.addresses:                         --
ipv6.gateway:                           --
ipv6.routes:                            --
ipv6.route-metric:                      -1
ipv6.route-table:                       0 (unspec)
ipv6.routing-rules:                     --
ipv6.ignore-auto-routes:                no
ipv6.ignore-auto-dns:                   no
ipv6.never-default:                     no
ipv6.may-fail:                          yes
ipv6.ip6-privacy:                       -1 (unknown)
ipv6.addr-gen-mode:                     stable-privacy
ipv6.ra-timeout:                        0 (default)
ipv6.dhcp-duid:                         --
ipv6.dhcp-iaid:                         --
ipv6.dhcp-timeout:                      0 (default)
ipv6.dhcp-send-hostname:                yes
ipv6.dhcp-hostname:                     --
ipv6.dhcp-hostname-flags:               0x0 (none)
ipv6.token:                             --
proxy.method:                           none
proxy.browser-only:                     no
proxy.pac-url:                          --
proxy.pac-script:                       --
GENERAL.NAME:                           Wired connection 3
GENERAL.UUID:                           f4b3782e-a253-3d50-828f-9936798f6f41
GENERAL.DEVICES:                        enp5s0
GENERAL.IP-IFACE:                       enp5s0
GENERAL.STATE:                          activated
GENERAL.DEFAULT:                        no
GENERAL.DEFAULT6:                       no
GENERAL.SPEC-OBJECT:                    --
GENERAL.VPN:                            no
GENERAL.DBUS-PATH:                      /org/freedesktop/NetworkManager/ActiveConnection/12
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/Settings/5
GENERAL.ZONE:                           --
GENERAL.MASTER-PATH:                    --
IP4.ADDRESS[1]:                         192.168.200.78/24
IP4.GATEWAY:                            192.168.200.254
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.200.254, mt = 102
IP4.ROUTE[2]:                           dst = 192.168.200.0/24, nh = 0.0.0.0, mt = 102
IP4.DNS[1]:                             192.168.200.253
DHCP4.OPTION[1]:                        dhcp_lease_time = 86400
DHCP4.OPTION[2]:                        dhcp_server_identifier = 192.168.200.254
DHCP4.OPTION[3]:                        domain_name_servers = 192.168.200.253
DHCP4.OPTION[4]:                        expiry = 1632386625
DHCP4.OPTION[5]:                        ip_address = 192.168.200.78
DHCP4.OPTION[6]:                        requested_broadcast_address = 1
DHCP4.OPTION[7]:                        requested_domain_name = 1
DHCP4.OPTION[8]:                        requested_domain_name_servers = 1
DHCP4.OPTION[9]:                        requested_domain_search = 1
DHCP4.OPTION[10]:                       requested_host_name = 1
DHCP4.OPTION[11]:                       requested_interface_mtu = 1
DHCP4.OPTION[12]:                       requested_ms_classless_static_routes = 1
DHCP4.OPTION[13]:                       requested_nis_domain = 1
DHCP4.OPTION[14]:                       requested_nis_servers = 1
DHCP4.OPTION[15]:                       requested_ntp_servers = 1
DHCP4.OPTION[16]:                       requested_rfc3442_classless_static_routes = 1
DHCP4.OPTION[17]:                       requested_root_path = 1
DHCP4.OPTION[18]:                       requested_routers = 1
DHCP4.OPTION[19]:                       requested_static_routes = 1
DHCP4.OPTION[20]:                       requested_subnet_mask = 1
DHCP4.OPTION[21]:                       requested_time_offset = 1
DHCP4.OPTION[22]:                       requested_wpad = 1
DHCP4.OPTION[23]:                       routers = 192.168.200.254
DHCP4.OPTION[24]:                       subnet_mask = 255.255.255.0
IP6.ADDRESS[1]:                         fe80::fa22:88ee:e229:6c83/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 102
Comment 15 Laurent Vivier 2021-09-23 08:31:57 UTC 
(In reply to Yanhui Ma from comment #14)
> (In reply to Laurent Vivier from comment #13)
> > (In reply to Yanhui Ma from comment #12)
> > > (In reply to Laurent Vivier from comment #11)
> > > > (In reply to Yanhui Ma from comment #9)
> > > > > The issue can be easily reproduced by live migration between rhel8.4.0 and
> > > > > rhel8.5.0, almost 100% reproduce ratio.
> > > > > 
> > > > > For steps and package version, pls refer to
> > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1999927#c0, just change win10
> > > > > guest to rhel8.4.0 guest.
> > > > > 
> > > > > # ifconfig 
> > > > > enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> > > > >         inet 10.73.74.5  netmask 255.255.252.0  broadcast 10.73.75.255
> > > > >         inet6 2620:52:0:4948:f68e:38ff:fec3:8080  prefixlen 64  scopeid
> > > > > 0x0<global>
> > > > >         inet6 fe80::f68e:38ff:fec3:8080  prefixlen 64  scopeid 0x20<link>
> > > > >         ether f4:8e:38:c3:80:80  txqueuelen 1000  (Ethernet)
> > > > >         RX packets 7582  bytes 548637 (535.7 KiB)
> > > > >         RX errors 0  dropped 169  overruns 0  frame 0
> > > > >         TX packets 151  bytes 15631 (15.2 KiB)
> > > > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > > > 
> > > > > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> > > > >         inet 192.168.200.222  netmask 255.255.255.0  broadcast
> > > > > 192.168.200.255
> > > > >         inet6 fe80::e9e9:9c38:3fb5:bd29  prefixlen 64  scopeid 0x20<link>
> > > > >         inet6 2001::6398:cd12:a87c:754c  prefixlen 64  scopeid 0x0<global>
> > > > >         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
> > > > >         RX packets 126  bytes 15440 (15.0 KiB)
> > > > >         RX errors 0  dropped 0  overruns 0  frame 0
> > > > >         TX packets 44  bytes 7372 (7.1 KiB)
> > > > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > > > 
> > > > > enp4s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> > > > >         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
> > > > >         RX packets 198  bytes 23023 (22.4 KiB)
> > > > >         RX errors 0  dropped 0  overruns 0  frame 0
> > > > >         TX packets 52  bytes 9280 (9.0 KiB)
> > > > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > > > 
> > > > > enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> > > > >         inet 192.168.200.223  netmask 255.255.255.0  broadcast
> > > > > 192.168.200.255
> > > > >         inet6 fe80::9c3c:b999:e0b6:65f3  prefixlen 64  scopeid 0x20<link>
> > > > >         ether 52:54:00:aa:8c:ef  txqueuelen 1000  (Ethernet)
> > > > >         RX packets 8  bytes 1640 (1.6 KiB)
> > > > >         RX errors 0  dropped 0  overruns 0  frame 0
> > > > >         TX packets 4  bytes 1112 (1.0 KiB)
> > > > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > > 
> > > > Could you check you DON'T have the file
> > > > /etc/sysconfig/network-scripts/ifcfg-enp5s0 ?
> > > > 
> > > 
> > > Yes, only ifcfg-enp1s0 file exists.
> > > 
> > > 
> > > # cd /etc/sysconfig/network-scripts/
> > > # ls
> > > ifcfg-enp1s0
> > > # cat ifcfg-enp1s0 
> > > TYPE=Ethernet
> > > PROXY_METHOD=none
> > > BROWSER_ONLY=no
> > > BOOTPROTO=dhcp
> > > DEFROUTE=yes
> > > IPV4_FAILURE_FATAL=no
> > > IPV6INIT=yes
> > > IPV6_AUTOCONF=yes
> > > IPV6_DEFROUTE=yes
> > > IPV6_FAILURE_FATAL=no
> > > NAME=enp1s0
> > > UUID=d7c9441b-de89-4b4f-84c1-69b88ff06c38
> > > DEVICE=enp1s0
> > > ONBOOT=yes
> > > > Thanks
> > 
> > 
> > This is really weird. Who configure enp4s0 and enp5s0 then?
> > 
> 
> Hi Laurent,
> 
> Please note the additional info in comment 0, my environment is the same
> with that.
> (3) The PF is connected to a private switch that I have configured with DHCP
> and VLAN services.
> The switch configure details is in the attachment.
> 
> 
> 
> > Do you have results for "nmcli con show enp4s0 enp5s0"?
> 
> # nmcli c show
> NAME                UUID                                  TYPE      DEVICE  
> 
> Wired connection 2  52fbcc8b-0f0f-3553-a693-aa3ab0052240  ethernet 
> enp4s0nsby 
> enp1s0              d7c9441b-de89-4b4f-84c1-69b88ff06c38  ethernet  enp1s0  
> 
> Wired connection 1  b8269223-09ba-3433-b75f-638e3f87d4c7  ethernet  enp4s0  
> 
> Wired connection 3  f4b3782e-a253-3d50-828f-9936798f6f41  ethernet  enp5s0
> 

So IP addresses are assigned by the configuration defined in NetworkManager. There is nothing we can do at qemu or kernel level to prevent that.

If you didn't configure manually these connections I think there is a bug somewhere in NetworkManager or in some configuration scripts.

Perhaps "journalctl -t NetworkManager" can give some hints.

Perhaps you can avoid the problem by adding enp4s0nsby and enp5s0 to the NetworkManager unmanaged device list?

cat > /etc/NetworkManager/conf.d/99-unmanaged-devices.conf <<EOF
[keyfile]
unmanaged-devices=interface-name:enp4s0nsby;interface-name:enp5s0
EOF

systemctl reload NetworkManager

nmcli device status
Comment 16 Yanhui Ma 2021-09-26 08:21:30 UTC 
(In reply to Laurent Vivier from comment #15)

> > 
> > > Do you have results for "nmcli con show enp4s0 enp5s0"?
> > 
> > # nmcli c show
> > NAME                UUID                                  TYPE      DEVICE  
> > 
> > Wired connection 2  52fbcc8b-0f0f-3553-a693-aa3ab0052240  ethernet 
> > enp4s0nsby 
> > enp1s0              d7c9441b-de89-4b4f-84c1-69b88ff06c38  ethernet  enp1s0  
> > 
> > Wired connection 1  b8269223-09ba-3433-b75f-638e3f87d4c7  ethernet  enp4s0  
> > 
> > Wired connection 3  f4b3782e-a253-3d50-828f-9936798f6f41  ethernet  enp5s0
> > 
> 
> So IP addresses are assigned by the configuration defined in NetworkManager.
> There is nothing we can do at qemu or kernel level to prevent that.
> 
> If you didn't configure manually these connections I think there is a bug
> somewhere in NetworkManager or in some configuration scripts.
> 
> Perhaps "journalctl -t NetworkManager" can give some hints.

Yes, you are right. There are indeed two IP assignment for enp5s0 in the output of "journalctl -t NetworkManager".
And I didn't configure connections manually.

Sep 25 17:43:07 vm-74-31.lab.eng.pek2.redhat.com NetworkManager[860]: <info>  [1632562987.5009] dhcp4 (enp5s0): state changed unknown -> bound, address=192.168.200.117
Sep 25 17:43:46 vm-74-31.lab.eng.pek2.redhat.com NetworkManager[860]: <info>  [1632563026.8044] dhcp4 (enp5s0): state changed unknown -> bound, address=192.168.200.181

I have attached the whole output.

> 
> Perhaps you can avoid the problem by adding enp4s0nsby and enp5s0 to the
> NetworkManager unmanaged device list?
> 
> cat > /etc/NetworkManager/conf.d/99-unmanaged-devices.conf <<EOF
> [keyfile]
> unmanaged-devices=interface-name:enp4s0nsby;interface-name:enp5s0
> EOF
> 
> systemctl reload NetworkManager
> 
> nmcli device status

Yes, after this, the enp5s0 will not be assigned IP.
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.74.31  netmask 255.255.252.0  broadcast 10.73.75.255
        inet6 fe80::5054:ff:fe00:8585  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:00:85:85  txqueuelen 1000  (Ethernet)
        RX packets 919  bytes 66420 (64.8 KiB)
        RX errors 0  dropped 21  overruns 0  frame 0
        TX packets 122  bytes 14507 (14.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.200.115  netmask 255.255.255.0  broadcast 192.168.200.255
        inet6 2001::aa6f:7c28:3f56:1bb  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::5a73:907e:2c13:e696  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 54  bytes 6249 (6.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 47  bytes 5002 (4.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp4s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::5054:ff:feaa:1cef  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 87  bytes 10531 (10.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 13  bytes 1400 (1.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::5054:ff:feaa:1cef  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:aa:1c:ef  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


So now I'd like to confirm with you I should file a bug for NetworkManager, right?
Comment 17 Laurent Vivier 2021-09-27 07:23:45 UTC 
(In reply to Yanhui Ma from comment #16)
...
> So now I'd like to confirm with you I should file a bug for NetworkManager,
> right?

You can re-assign this bug to NetworkManager component to have at least an analysis of the cause of the problem.

After that we will know if the problem comes from NM or from something in the kernel or in the OS.
Comment 18 Yanhui Ma 2021-09-27 09:09:21 UTC 
(In reply to Laurent Vivier from comment #17)
> (In reply to Yanhui Ma from comment #16)
> ...
> > So now I'd like to confirm with you I should file a bug for NetworkManager,
> > right?
> 
> You can re-assign this bug to NetworkManager component to have at least an
> analysis of the cause of the problem.
> 
> After that we will know if the problem comes from NM or from something in
> the kernel or in the OS.
Done, thank you very much for your all analysis.