Description of problem: virsh net-dumpxml baremetal-0 <network connections='1' ipv6='yes'> <name>baremetal-0</name> <uuid>669c4383-d4c1-41c3-bfc0-6ec166c3c973</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='baremetal-0' stp='on' delay='0'/> <mac address='52:54:00:cb:80:fc'/> <domain name='ocp-edge-cluster-0.qe.lab.redhat.com' localOnly='yes'/> <dns> <forwarder domain='apps.ocp-edge-cluster-0.qe.lab.redhat.com' addr='127.0.0.1'/> <host ip='192.168.123.5'> <hostname>api</hostname> </host> <host ip='192.168.123.1'> <hostname>registry</hostname> <hostname>hypervisor</hostname> </host> </dns> <ip address='192.168.123.1'> <dhcp> <range start='192.168.123.100' end='192.168.123.150'> <lease expiry='0'/> </range> </dhcp> </ip> </network> virsh net-dhcp-leases baremetal-0 error: Failed to get leases info for baremetal-0 error: internal error: found lease without expiry-time Version-Release number of selected component (if applicable): libvirt-6.6.0-6.module+el8.3.0+8125+aefcf088.x86_64 How reproducible: 100% Steps to Reproduce: 1. Create network with <lease expiry='0'/> 2. Create VM connected to network 3. Run virsh net-dhcp-leases Actual results: error: Failed to get leases info for baremetal-0 error: internal error: found lease without expiry-time Expected results: Can retrieve leases even if they have infinite allocation. Additional info:
I can reproduce it on libvirt-libs-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64
Not only "virsh net-dhcp-leases" is broken, it's also NSS plugin and "domifaddr --source lease". Patches posted upstream: https://www.redhat.com/archives/libvir-list/2020-December/msg00811.html
Test on upstream, the mac address from the net-dhcp-leases is not correct, please check it: ➜ ~ virsh domiflist rhel Interface Type Source Model MAC ------------------------------------------------------------- vnet14 network default virtio 52:54:00:5c:af:6c ➜ ~ virsh net-dhcp-leases default Expiry Time MAC address Protocol IP address Hostname Client ID or DUID ------------------------------------------------------------------------------------------------------------ 1970-01-01 00:00:00 52:54:00:5c:af:6c ipv4 192.168.122.245/24 test **** 01:52:54:00:5c:af:6c ****
After libvirtd restart, the hostname of the guest in one of the leases disappear, please help to check, Thank you! ➜ ~ virsh net-dumpxml default <network connections='2'> <name>default</name> <uuid>7dbd7798-b46f-4044-bd8d-bf11e3c31cb4</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr0' stp='on' delay='0'/> <mac address='52:54:00:ea:c3:cf'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'> <lease expiry='0'/> </range> <host mac='52:16:3e:77:e2:ed' ip='192.168.122.12'> <lease expiry='1' unit='hours'/> </host> </dhcp> </ip> </network> ➜ ~ virsh net-dhcp-leases default Expiry Time MAC address Protocol IP address Hostname Client ID or DUID ------------------------------------------------------------------------------------------------------------ 2020-12-22 11:22:36 52:16:3e:77:e2:ed ipv4 192.168.122.12/24 test 01:52:16:3e:77:e2:ed 1970-01-01 00:00:00 52:54:00:6d:97:24 ipv4 192.168.122.188/24 test 01:52:54:00:6d:97:24 ➜ ~ cat /var/lib/libvirt/dnsmasq/virbr0.status [ { "ip-address": "192.168.122.188", "mac-address": "52:54:00:6d:97:24", "hostname": "test", "client-id": "01:52:54:00:6d:97:24", "expiry-time": 0 }, { "ip-address": "192.168.122.12", "mac-address": "52:16:3e:77:e2:ed", "hostname": "test", "client-id": "01:52:16:3e:77:e2:ed", "expiry-time": 1608636156 } ] ➜ ~ systemctl restart libvirtd ➜ ~ cat /var/lib/libvirt/dnsmasq/virbr0.status [ { "ip-address": "192.168.122.12", "mac-address": "52:16:3e:77:e2:ed", "hostname": "test", "client-id": "01:52:16:3e:77:e2:ed", "expiry-time": 1608636156 }, { "ip-address": "192.168.122.188", "mac-address": "52:54:00:6d:97:24", "client-id": "01:52:54:00:6d:97:24", ***** after restart libvirtd, the hostname disappear in the status file "expiry-time": 0 } ] ➜ ~ virsh net-dhcp-leases default Expiry Time MAC address Protocol IP address Hostname Client ID or DUID ------------------------------------------------------------------------------------------------------------ 2020-12-22 11:22:36 52:16:3e:77:e2:ed ipv4 192.168.122.12/24 test 01:52:16:3e:77:e2:ed ** 1970-01-01 00:00:00 52:54:00:6d:97:24 ipv4 192.168.122.188/24 - 01:52:54:00:6d:97:24 ***after restart libvirtd, the hostname disappear
(In reply to yalzhang from comment #3) > Test on upstream, the mac address from the net-dhcp-leases is not correct, > please check it: > > ➜ ~ virsh domiflist rhel > Interface Type Source Model MAC > ------------------------------------------------------------- > vnet14 network default virtio 52:54:00:5c:af:6c > > ➜ ~ virsh net-dhcp-leases default > Expiry Time MAC address Protocol IP address > Hostname Client ID or DUID > ----------------------------------------------------------------------------- > ------------------------------- > 1970-01-01 00:00:00 52:54:00:5c:af:6c ipv4 192.168.122.245/24 > test **** 01:52:54:00:5c:af:6c **** I think this is in fact correct. MAC address is the same for both commands. And this Client ID is NOT a MAC address rather than unique/random generated bytes that dhcp client sends during DHCP transaction. In other words, this is expected. https://tools.ietf.org/html/rfc2132#section-9.14
(In reply to yalzhang from comment #4) > After libvirtd restart, the hostname of the guest in one of the leases > disappear, please help to check, Thank you! > ➜ ~ virsh net-dhcp-leases default > Expiry Time MAC address Protocol IP address > Hostname Client ID or DUID > ----------------------------------------------------------------------------- > ------------------------------- > 2020-12-22 11:22:36 52:16:3e:77:e2:ed ipv4 192.168.122.12/24 > test 01:52:16:3e:77:e2:ed > ** 1970-01-01 00:00:00 52:54:00:6d:97:24 ipv4 192.168.122.188/24 > - 01:52:54:00:6d:97:24 ***after restart libvirtd, the hostname > disappear I don't think this is linked to my patches but to the fact that you used the same hostname twice. I wasn't able to reproduce using different hostnames.
(In reply to Michal Privoznik from comment #6) > (In reply to yalzhang from comment #4) > > After libvirtd restart, the hostname of the guest in one of the leases > > disappear, please help to check, Thank you! > > > ➜ ~ virsh net-dhcp-leases default > > Expiry Time MAC address Protocol IP address > > Hostname Client ID or DUID > > ----------------------------------------------------------------------------- > > ------------------------------- > > 2020-12-22 11:22:36 52:16:3e:77:e2:ed ipv4 192.168.122.12/24 > > test 01:52:16:3e:77:e2:ed > > ** 1970-01-01 00:00:00 52:54:00:6d:97:24 ipv4 192.168.122.188/24 > > - 01:52:54:00:6d:97:24 ***after restart libvirtd, the hostname > > disappear > > I don't think this is linked to my patches but to the fact that you used the > same hostname twice. I wasn't able to reproduce using different hostnames. The 2 interfaces are attached to the same guest, check the guest xml as below: # virsh dumpxml rhel | grep /interface -B9 <interface type='network'> <mac address='52:16:3e:77:e2:ed'/> <source network='default' portid='458dcf8b-4af8-4c10-9f67-411b8643c2e4' bridge='virbr0'/> <target dev='vnet35'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:6d:97:24'/> <source network='default' portid='bb25e84d-d409-43e8-addb-d9cb9a97efdb' bridge='virbr0'/> <target dev='vnet36'/> <model type='virtio'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </interface>
(In reply to yalzhang from comment #7) > > The 2 interfaces are attached to the same guest, check the guest xml as > below: Okay, I am able to reproduce now. But even without defining lease expiry time, that is with the following network XML: <network connections='2'> <name>default</name> <uuid>4a446508-5f7f-433e-a0c2-b66dcbec4886</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr0' stp='on' delay='0'/> <mac address='52:54:00:ad:bd:ff'/> <bandwidth> <inbound average='4096'/> <outbound average='4096'/> </bandwidth> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> <host mac='52:54:00:a4:6f:91' name='fedora' ip='192.168.122.3'/> </dhcp> </ip> </network> Hence, I don't think this is related to my patches. It's still worth fixing. So far my debugging lead me to dnsmasq. When libvirtd is restarted then the leases helper is invoked multiple times and that's what changing the virbr0.status file. We have to go deeper.
Alright, so what's happening is: when libvirtd starts up, networkRefreshDhcpDaemon() is called (because the network is already running). This rewrites .hostsfile and then sends SIGHUP to the dnsmasq to reload configuration. In turn, dnsmasq invokes our leases helper, trying to inform us that configuration has changed. These are the invocations I've observed: 865 ? S 0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/home/zippy/work/libvirt/libvirt.git/_build/src/libvirt_leaseshelper 866 ? t 0:00 \_ /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/home/zippy/work/libvirt/libvirt.git/_build/src/libvirt_leaseshelper 4302 ? t 0:00 \_ libvirt_leaseshelper old 52:54:00:eb:ab:db 192.168.122.145 865 ? S 0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/home/zippy/work/libvirt/libvirt.git/_build/src/libvirt_leaseshelper 866 ? t 0:00 \_ /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/home/zippy/work/libvirt/libvirt.git/_build/src/libvirt_leaseshelper 4537 ? t 0:00 \_ libvirt_leaseshelper old 52:54:00:a4:6f:91 192.168.122.3 fedora But as we can see the hostname was provided only for the second invocation. The hostname was not provided even through an environment variable. Therefore, leaseshelper removed it from the file. We can open a bug against dnsmasq, if anything.
Hi Michal, Thank you for the detailed analysis. File Bug 1910621 against dnsmasq for issue in comment 4, 6~9
Merged upstream as: 7f93905e45 nss: handle leases with infinite expiry time 5dd53684e1 networkGetDHCPLeases: Handle leases with infinite expiry time 5fb6d98c88 network: Rework networkGetDHCPLeases() ee93656c40 networkGetDHCPLeases: Use VIR_APPEND_ELEMENT() instead of VIR_INSERT_ELEMENT() 9c65363a40 network: Drop @custom_lease_file_len variable from networkGetDHCPLeases() 6f1ae57129 virlease: Allow infinite lease expiry time 003fff38e7 virlease: Use virTrimSpaces() instead of open coded alternative 8e5659ed12 virlease: Rework virLeaseReadCustomLeaseFile() c14bd64f3e leaseshelper: Report errors on failure 49869e8d57 docs: Document ability to configure lease time v6.10.0-281-g7f93905e45
Moving to POST per comment 11.
test on libvirt-7.0.0-1.module+el8.4.0+9464+3e71831a.x86_64, the bug is fixed. # virsh net-dumpxml test5 <network connections='1'> <name>test5</name> <uuid>9d0ddef3-0ce9-4d1e-ad24-be10477f0e79</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr1' stp='on' delay='0'/> <mac address='52:54:00:cf:d0:48'/> <ip address='192.168.232.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.232.2' end='192.168.232.254'> <lease expiry='0'/> </range> <host mac='00:16:3e:77:e2:ed' name='bar1.example.com' ip='192.168.232.10'> <lease expiry='0'/> </host> <host mac='00:16:3e:77:e2:aa' name='foo1.example.com' ip='192.168.232.11'> <lease expiry='0'/> </host> </dhcp> </ip> </network> # virsh net-dhcp-leases test5 Expiry Time MAC address Protocol IP address Hostname Client ID or DUID ----------------------------------------------------------------------------------------------------------- 1969-12-31 19:00:00 52:54:00:2b:8f:ec ipv4 192.168.232.12/24 - 01:52:54:00:2b:8f:ec # virsh attach-interface test network test5 --mac 00:16:3e:77:e2:aa --model virtio Interface attached successfully # virsh net-dhcp-leases test5 Expiry Time MAC address Protocol IP address Hostname Client ID or DUID ----------------------------------------------------------------------------------------------------------- 1969-12-31 19:00:00 00:16:3e:77:e2:aa ipv4 192.168.232.11/24 foo1 01:00:16:3e:77:e2:aa 1969-12-31 19:00:00 52:54:00:2b:8f:ec ipv4 192.168.232.12/24 - 01:52:54:00:2b:8f:ec
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098