Description of problem: Restart libvirtd will clear the network connection count Version-Release number of selected component (if applicable): libvirt-5.6.0-2.virtcov.el8.x86_64 How reproducible: 100% Steps to Reproduce: 1. Prepare different type of networks including: nat, bridge type with shared host bridge, macvtap, hostdev, bridge type with ovs bridge; # virsh net-list --all Name State Autostart Persistent --------------------------------------------------- bridge-test active no yes default active yes yes direct-macvtap active no yes ovs-net active no yes passthrough1 active no yes 2. Attached 1 interface from each network to a running vm, check the connection of the networks, all of them got the connections='1', which is expected: # virsh net-dumpxml bridge-test | grep connection <network connections='1'> # virsh net-dumpxml default | grep connection <network connections='1'> # virsh net-dumpxml direct-macvtap | grep connection <network connections='1'> <interface dev='eno25' connections='1'/> # virsh net-dumpxml ovs-net | grep connection <network connections='1'> # virsh net-dumpxml passthrough1 | grep connection <network connections='1'> 3. Restart libvirtd and check the connection again, both of the connection='1' will gone: # systemctl restart libvirtd # virsh net-dumpxml passthrough1 | grep "<network" -A1 <network> <name>passthrough1</name> # virsh net-dumpxml bridge-test | grep "<network" -A1 <network> <name>bridge-test</name> ...... Actual results: Restart libvirtd will clear the network connection count Expected results: The connection count should not change after libvirtd restart Additional info: For direct type passthrough mode, there is an additional issue related with this one: 1. Prepare a running guest with 1 interface from direct type passthrough mode network: # virsh dumpxml rhel | grep /interface -B7 <interface type='direct'> <mac address='52:54:00:b9:fb:d8'/> <source network='direct-macvtap' portid='09349016-4503-45f2-8c45-1c3e27b72248' dev='eno25' mode='passthrough'/> <target dev='macvtap0'/> <model type='rtl8139'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x0a' slot='0x01' function='0x0'/> </interface> # virsh net-dumpxml direct-macvtap <network connections='1'> <name>direct-macvtap</name> <uuid>ed1aca1f-cc1a-4e5b-b9f4-fb617134ea2d</uuid> <forward dev='eno25' mode='passthrough'> <interface dev='eno25' connections='1'/> </forward> </network> 2. Restart libvirtd: # systemctl restart libvirtd 3. check the connection is gone as this bug: # virsh net-dumpxml direct-macvtap <network> <name>direct-macvtap</name> <uuid>ed1aca1f-cc1a-4e5b-b9f4-fb617134ea2d</uuid> <forward dev='eno25' mode='passthrough'> <interface dev='eno25'/> </forward> </network> 4. Destroy the vm # virsh destroy rhel Domain rhel destroyed 5. Try to start it again, it will fail # virsh dumpxml rhel | grep /interface -B5 <interface type='network'> <mac address='52:54:00:b9:fb:d8'/> <source network='direct-macvtap'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x0a' slot='0x01' function='0x0'/> </interface> # virsh start rhel error: Failed to start domain rhel error: internal error: network 'direct-macvtap' requires exclusive access to interfaces, but none are available After the vm is destroyed, the interface should be free, and in step 5, it should start successfully.
git bisect shows the lost connection count is caused by commit fe7719730419ce04ebaeeb8aa3dadd28c9554cf8 Author: Daniel P. Berrangé <berrange> Date: Mon Dec 24 10:45:00 2018 +0000 conf: switch over to use network port APIs for virt drivers Change the domain conf so invoke the new network port public APIs instead of the network callbacks. Reviewed-by: Laine Stump <laine> Signed-off-by: Daniel P. Berrangé <berrange>
Patch proposed upstream https://www.redhat.com/archives/libvir-list/2019-September/msg00558.html
Merged upstream with commit 7ab9bdd470ea3325da9101f9cce2d0ffbb10ac41 Author: Daniel P. Berrangé <berrange> Date: Fri Sep 13 15:54:18 2019 +0100 network: fix connection usage counts after restart Since the introduction of the virNetworkPort object, the network driver has a persistent record of ports that have been created against the networks. Thus the hypervisor drivers no longer communicate to the network driver during libvirtd restart. This change, however, meant that the connection usage counts were no longer re-initialized during a libvirtd restart. To deal with this we must iterate over all virNetworkPortDefPtr objects we have and invoke the notify callback to record the connection usage count. Reviewed-by: Laine Stump <laine> Signed-off-by: Daniel P. Berrangé <berrange>
These patches fix the case when a network is destroyed/re-started and then libvirtd is restarted: https://www.redhat.com/archives/libvir-list/2019-September/msg00900.html
Re: patches posted in Comment 4 - slightly modified versions were pushed upstream as: Author: Laine Stump <laine> Date: Sun Sep 22 21:03:51 2019 -0400 conf: take advantage of VIR_AUTOPTR for virNetworkPortDefPtr (this is just a prerequisite cleanup for the fix in the following patch) commit 98fe739e37b0b187e9ae777177124d507e8daac1 Author: Laine Stump <laine> Date: Fri Sep 20 18:15:19 2019 -0400 conf: refresh network ports missing from network driver on restart Any branch that needs to backport to fix this bug will need to backport both of those patches, as well as the patch Daniel mentioned in Comment 3.
Verified it with libvirt-6.0.0-1.virtcov.el8.x86_64 & qemu-kvm-4.2.0-6.module+el8.2.0+5453+31b2b136.x86_64 1. Prepare different type of networks # virsh net-list --all Name State Autostart Persistent ----------------------------------------------------- bridge inactive no yes default active yes yes direct-macvtap inactive no yes ovs-net active no yes passthrough1 active no yes 2. Attached one interface from each network to a running vm, check the connection of the networks, all of them got the connections='1': # virsh net-dumpxml bridge | grep connection <network connections='1'> # virsh net-dumpxml default | grep connection <network connections='1'> # virsh net-dumpxml direct-macvtap | grep connection <network connections='1'> <interface dev='eno25' connections='1'/> # virsh net-dumpxml ovs-net | grep connection <network connections='1'> # virsh net-dumpxml passthrough1 | grep connection <network connections='1'> 3. Restart libvirtd and check the connection again, the connection='1' is still exist for all the network. # systemctl restart libvirtd # virsh net-dumpxml passthrough1 | grep "<network" -A1 <network connections='1'> <name>passthrough1</name> # virsh net-dumpxml bridge | grep "<network" -A1 <network connections='1'> <name>bridge-test</name> ....
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2017