Description of problem: systemctl restart libvirtd.service results in: Jul 01 03:38:38 systemd[1]: Stopping Virtualization daemon... Jul 01 03:38:46 systemd[1]: Stopped Virtualization daemon. Jul 01 03:38:46 systemd[1]: libvirtd.service: Found left-over process 11895 (dns masq) in control group while starting unit. Ignoring. Jul 01 03:38:46 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Jul 01 03:38:46 systemd[1]: libvirtd.service: Found left-over process 11896 (dnsmasq) in control group while starting unit. Ignoring. Jul 01 03:38:46 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Jul 01 03:38:46 systemd[1]: Starting Virtualization daemon... Jul 01 03:38:48 systemd[1]: Started Virtualization daemon. # ps -fe | grep dnsma dnsmasq 11895 1 0 Jun25 ? 00:00:10 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper root 11896 11895 0 Jun25 ? 00:00:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper Version-Release number of selected component (if applicable): libvirt-4.1.0-3.fc28.x86_64 How reproducible: Everytime Steps to Reproduce: 1. systemctl restart libvirtd 2. See it dnsmasq has been restarted Actual results: dnsmasq has not been restarted Expected results: dnsmasq was restarted
In this case, process 11896 was not responding to SIGTERM or SIGHUP, had to use SIGKILL.
This is intentional. All running virtual machines must remain uninterrupted when libvirtd is stopped or restarted. dnsmasq provides an ongoing service to VMs, so dnsmasq is also left running.
I don't understand why dnsmasq cannot continue to provide those service after restart. If we need to restart dnsmasq due to system updates, how are we supposed to do this?
There no notion of a service "restart" in systemd - it is just a combination of a "stop" followed by a "start". We do not wish to terminate service to VMs during a normal "stop", as there can be a arbitrary delay before the admin does a "start" again during which times VMs will be impacted. The only way to restart dnsmasq is to stop+start the virtual network associated with it using virsh
Would it be possible for libvirt to start the dnsmasq process via a separate systemd service unit? We use a script to restart certain processes automatically via their associated service unit when needed due to updates. Currently: # ps -fe | grep dnsmas nobody 2132 1 0 07:10 ? 00:00:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper root 2133 2132 0 07:10 ? 00:00:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper # systemctl status 2133 ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2018-07-02 07:10:50 MDT; 2h 18min ago Docs: man:libvirtd(8) https://libvirt.org Main PID: 1442 (libvirtd) Tasks: 19 (limit: 32768) CGroup: /system.slice/libvirtd.service ├─1442 /usr/sbin/libvirtd ├─2132 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefil... └─2133 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefil... associates the dnsmasq process with libvirtd.service, suggesting that restarting it would restart dnsmasq. If the processes were attached to a libvirtd-dnsmasq.service unit, they could be restarted independently through the normal systemd channels.
Yes, it would be possible to dynamically create systemd units for the dnsmasq processes we spawn, when running on systemd enabled hosts. I can't promise anyone will work on that any time in the near future though. We'd take patches if someone wants to try implementing it themselves any time.
Stopping libvirt service doesn't stop dnsmasq either: # systemctl stop libvirtd # systemctl status libvirtd ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: inactive (dead) since Thu 2019-02-28 00:15:46 CET; 1s ago Docs: man:libvirtd(8) https://libvirt.org Process: 11201 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS) Main PID: 11201 (code=exited, status=0/SUCCESS) Tasks: 2 (limit: 32768) Memory: 57.5M CGroup: /system.slice/libvirtd.service ├─16839 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/us> └─16840 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/us> Feb 28 00:15:46 localhost.localdomain systemd[1]: Stopping Virtualization daemon... Feb 28 00:15:46 localhost.localdomain systemd[1]: Stopped Virtualization daemon. This is on F29: libvirt-daemon-4.7.0-1.fc29.x86_64(In reply to Daniel Berrange from comment #4) > The only way to restart dnsmasq is to stop+start the virtual network > associated with it using virsh ...which you can't do if libvirt is not running: # virsh net-list error: failed to connect to the hypervisor error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
That is by design. Stopping libvirtd should _not_ kill your VMs and the networks which they require. Otherwise the VMs would not survive an upgrade of the system or a crash of libvirtd. This means you have to restart the network using the virsh API when libvirtd is running. Please note that restarting a network while VMs are using it leads to loss of connectivity for said VMs.
(In reply to Peter Krempa from comment #8) > That is by design. When stopping any service, it is expected that all processes associated with it will be stopped. Otherwise it's a bug. Could you reopen this? > Stopping libvirtd should _not_ kill your VMs and the networks which they > require. Otherwise the VMs would not survive an upgrade of the system or > a crash of libvirtd. > > This means you have to restart the network using the virsh API when > libvirtd is running. And if it isn't then I cannot restart the leftover dnsmasq processes gracefully, only by killing them by their PIDs. This is not a good design. If the dnsmasq process(es) should be managed independently of the libvirtd service then as Orion wrote, they should have their own systemd service unit. > Please note that restarting a network while VMs are using it > leads to loss of connectivity for said VMs. Granted, but that's expected.
(In reply to Dominik 'Rathann' Mierzejewski from comment #9) > (In reply to Peter Krempa from comment #8) > > That is by design. > > When stopping any service, it is expected that all processes associated > with it will be stopped. Otherwise it's a bug. Could you reopen this? No. As said above this would also mean that all qemu processes runnign the VM would be killed. That is against our design. Eventually we plan to add a better alternative of the libvirt-guests service which which will also start/restart/save/kill VMs at start and shutdown of the host. This could also eventually deal with networks. > > Stopping libvirtd should _not_ kill your VMs and the networks which they > > require. Otherwise the VMs would not survive an upgrade of the system or > > a crash of libvirtd. > > > > This means you have to restart the network using the virsh API when > > libvirtd is running. > > And if it isn't then I cannot restart the leftover dnsmasq processes > gracefully, only by killing them by their PIDs. > > This is not a good design. If the dnsmasq process(es) should be managed > independently of the libvirtd service then as Orion wrote, they should > have their own systemd service unit. If you wish to run your own managed networks via systemd you are welcome to do so by removing any libvirt managed networks and starting your own dnsmasq instance. Libvirt networks need to persist libvirtd restarts to keep connectivity for the VMs which also persist through libvirtd restarts.