Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1711180

Summary: vm connecting to the default virtual net will lose connection after restart the virtual network and restart libvirtd
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: yisun
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED CURRENTRELEASE QA Contact: yalzhang <yalzhang>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.1CC: chhu, dyuan, jdenemar, xuzhang, yalzhang, yisun
Target Milestone: rcKeywords: Automation, Regression, TestOnly
Target Release: 8.3Flags: pm-rhel: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-6.0.0-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-01-08 16:28:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description yisun 2019-05-17 07:21:44 UTC 
Description: vm connecting to the default virtual net will lose connection after restart the virtual network and restart libvirtd

Version:
libvirt-5.3.0-1.module+el8.1.0+3164+94495c71.x86_64

Reproduce:
100%

Key Words:
Regression, Automation
Not reproduced with libvirt-5.0.0-8.module+el8+3201+fd6ac632.x86_64

PLEASE NOTE, this is quite similar as bz1309529, but we add a step to restart libvirtd. For previous libvirt version, restart libvirtd will make the connection back.


Steps:
1. Having a vm connecting to the default virtual network, and we can PING to it
[root@dell-per740-18 ~]# virsh net-list
 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   yes         yes

[root@dell-per740-18 ~]# virsh net-dumpxml default
<network connections='1'>
  <name>default</name>
  <uuid>6cb5d138-cacd-4dda-84c0-45831d9c1d53</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:9d:24:7e'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

[root@dell-per740-18 ~]# virsh dumpxml gls | grep "<interface" -A7
    <interface type='network'>
      <mac address='52:54:77:77:77:77'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>


[root@dell-per740-18 ~]# virsh domif-getlink gls vnet0
vnet0 up

[root@dell-per740-18 ~]# virsh domifaddr gls
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet0      52:54:77:77:77:77    ipv4         192.168.122.99/24

[root@dell-per740-18 ~]# ping 192.168.122.99 -c 4
PING 192.168.122.99 (192.168.122.99) 56(84) bytes of data.
64 bytes from 192.168.122.99: icmp_seq=1 ttl=64 time=0.290 ms
64 bytes from 192.168.122.99: icmp_seq=2 ttl=64 time=0.272 ms
64 bytes from 192.168.122.99: icmp_seq=3 ttl=64 time=0.269 ms
64 bytes from 192.168.122.99: icmp_seq=4 ttl=64 time=0.270 ms
--- 192.168.122.99 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 78ms


2. Destroy and start the default virtual network, then restart libvirtd, as follow：
[root@dell-per740-18 ~]# virsh net-destroy default; virsh net-start default; service libvirtd restart; sleep 10
Network default destroyed

Network default started

Redirecting to /bin/systemctl restart libvirtd.service


3. Ping the vm again, connection lost
[root@dell-per740-18 ~]# virsh domif-getlink gls vnet0
vnet0 up

[root@dell-per740-18 ~]# virsh domifaddr gls
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet0      52:54:77:77:77:77    ipv4         192.168.122.99/24

[root@dell-per740-18 ~]# ping 192.168.122.99 -c 4
PING 192.168.122.99 (192.168.122.99) 56(84) bytes of data.
From 192.168.122.1 icmp_seq=1 Destination Host Unreachable
From 192.168.122.1 icmp_seq=2 Destination Host Unreachable
From 192.168.122.1 icmp_seq=3 Destination Host Unreachable
From 192.168.122.1 icmp_seq=4 Destination Host Unreachable
--- 192.168.122.99 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 66ms


Additional info:
Checked another auto job of rhel8.0.0, nothing wrong.
And we cannot manually reproduce this with libvirt-5.0.0-8.module+el8+3201+fd6ac632.x86_64, as follow:
[root@hp-dl320eg8-10 ~]# virsh domifaddr gls
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet0      52:54:77:77:77:77    ipv4         192.168.122.99/24

[root@hp-dl320eg8-10 ~]# ping -c 4 192.168.122.99
PING 192.168.122.99 (192.168.122.99) 56(84) bytes of data.
64 bytes from 192.168.122.99: icmp_seq=1 ttl=64 time=0.169 ms
64 bytes from 192.168.122.99: icmp_seq=2 ttl=64 time=0.208 ms
64 bytes from 192.168.122.99: icmp_seq=3 ttl=64 time=0.182 ms
64 bytes from 192.168.122.99: icmp_seq=4 ttl=64 time=0.177 ms

--- 192.168.122.99 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 97ms
rtt min/avg/max/mdev = 0.169/0.184/0.208/0.014 ms
[root@hp-dl320eg8-10 ~]# virsh net-destroy default; virsh net-start default; service libvirtd restart
Network default destroyed

Network default started

Redirecting to /bin/systemctl restart libvirtd.service
[root@hp-dl320eg8-10 ~]# ping -c 4 192.168.122.99
PING 192.168.122.99 (192.168.122.99) 56(84) bytes of data.
64 bytes from 192.168.122.99: icmp_seq=1 ttl=64 time=0.288 ms
64 bytes from 192.168.122.99: icmp_seq=2 ttl=64 time=0.155 ms
64 bytes from 192.168.122.99: icmp_seq=3 ttl=64 time=0.182 ms
64 bytes from 192.168.122.99: icmp_seq=4 ttl=64 time=0.163 ms

--- 192.168.122.99 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 58ms
rtt min/avg/max/mdev = 0.155/0.197/0.288/0.053 ms


Expected result:
After destroy/start network and restart libvirtd, the vm should back online.
Comment 2 yisun 2019-05-17 07:28:59 UTC 
Yalan Zhang found a clue about this:
https://patchew.org/Libvirt/20170321150400.13422-1-laine@laine.org/
Comment 3 yalzhang@redhat.com 2019-05-17 08:54:37 UTC 
(In reply to yisun from comment #2)
> Yalan Zhang found a clue about this:
> https://patchew.org/Libvirt/20170321150400.13422-1-laine@laine.org/

One addition: the above patch is not the one that cause this issue. It is to enable the feature 'reconnect tap devices after libvirtd restarted', but currently on libvirt-5.3.0-1, this feature is broken.
Comment 4 Laine Stump 2020-09-11 18:01:51 UTC 
Hmm. Somehow I managed to not update this BZ when this bug was fixed. It was fixed with a combination of:


commit 7ab9bdd470ea3325da9101f9cce2d0ffbb10ac41
Author: Daniel P. Berrang? <berrange>
Date:   Fri Sep 13 15:54:18 2019 +0100

    network: fix connection usage counts after restart

and

commit 98fe739e37b0b187e9ae777177124d507e8daac1
Author: Laine Stump <laine>
Date:   Fri Sep 20 18:15:19 2019 -0400

    conf: refresh network ports missing from network driver on restart

(both are in libvirt-5.8.0)

This means it was fixed by the recent rebase of libvirt.
Comment 6 Jiri Denemark 2020-09-11 19:00:23 UTC 
Hmm, the fixes were already released in 8.2.0 then (libvirt 6.0.0) so we could
close it right away, but I think letting QE verify the fix cannot hurt.
Comment 8 yalzhang@redhat.com 2020-09-15 01:45:08 UTC 
test on latest libvirt-6.6.0-4.module+el8.3.0+7883+3d717aa8.x86_64, the issue is fixed.
Comment 10 Jeff Nelson 2021-01-08 16:28:53 UTC 
RHEL AV 8.3.0 has been shipped, therefore marking this BZ CLOSED CURRENTRELEASE.