Whenever I reboot my machine and log in in the graphical interface I cannot run virtmanager, because it claims the libvirtd is not running. It's right. I can run it manually (as root) and then the virtmanager works like is used to in Fedora 24 before the upgrade of this machine to Fedora 25. This is quite low in the system for me, thus I asked for a help and I was told to run things like: # systemctl reenable libvirtd.service > Removed /etc/systemd/system/sockets.target.wants/virtlockd.socket. > Removed /etc/systemd/system/sockets.target.wants/virtlogd.socket. > Removed /etc/systemd/system/multi-user.target.wants/libvirtd.service. > Created symlink /etc/systemd/system/multi-user.target.wants/libvirtd.service → /usr/lib/systemd/system/libvirtd.service. > Created symlink /etc/systemd/system/sockets.target.wants/virtlockd.socket → /usr/lib/systemd/system/virtlockd.socket. > Created symlink /etc/systemd/system/sockets.target.wants/virtlogd.socket → /usr/lib/systemd/system/virtlogd.socket. And even the status looks fine: # systemctl status libvirtd.service > ● libvirtd.service - Virtualization daemon > Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) > Active: inactive (dead) > Docs: man:libvirtd(8) > http://libvirt.org but the process is still not running, only when I run it manually (as an old school): # service libvirtd start As I said above, it begun with Fedora 25, the previous Fedora 24 was all fine.
(In reply to Milan Crha from comment #0) > > Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) > > Active: inactive (dead) Just in case, this is the status after reboot. What I was referring to as "looks fine" was the "enabled|enabled" on the Loaded line.
I have also notice this, i.e. libvirtd started automatically under F24, but as soon as I moved to F25 it failed to automatically start on boot, even though it is enabled.
The same issue with libvirt-2.5.0-2.fc26.x86_64 on Fedora rawhide. Libvirtd.service is inactive after booting when it is enabled. But start it manually after booting is OK.
I was told that maybe Lukas will know, thus CC'ing him. Just in case. Thanks in advance.
If anyone is still hitting this, please provide the output of this command, after the machine has booted and libvirtd isn't running sudo journalctl --boot -u libvirtd.service
Sure, after reboot and login to a desktop environment, then opening a terminal and logging as root I also verified that libvirtd is not running and the output of the command you have is: [root@zyxPad mcrha]# ps ax | grep libvirtd 1710 pts/0 S+ 0:00 grep --color=auto libvirtd [root@zyxPad mcrha]# journalctl --boot -u libvirtd.service -- No entries --
Milan, indeed that's weird. I'd expect at least some kind of log message in journalctl since the service is listed as 'enabled' . Is there anything in 'journcalctl --boot | grep virt' ? Or maybe just attach the whole 'journalctl --boot' output after a reboot
# journalctl --boot | grep virt Mar 24 10:25:01 zyxPad kernel: Booting paravirtualized kernel on bare hardware Mar 24 09:25:13 zyxPad spice-vdagent[1406]: Cannot access vdagent virtio channel /dev/virtio-ports/com.redhat.spice.0 (It was 9:25AM here, not 10:25.)
(In reply to Milan Crha from comment #8) > # journalctl --boot | grep virt > Mar 24 10:25:01 zyxPad kernel: Booting paravirtualized kernel on bare > hardware > Mar 24 09:25:13 zyxPad spice-vdagent[1406]: Cannot access vdagent virtio > channel /dev/virtio-ports/com.redhat.spice.0 > > (It was 9:25AM here, not 10:25.) Hmm, haven't seen this first message before. Do you have xen installed? Not sure if it should make a difference though. Nothing really sticks out from the log though, so more info Can you also provide output of: * systemctl --no-pager * sudo find /etc -name \*libvirt\* * sudo rpm -q -V libvirt-daemon
Created attachment 1266577 [details] systemctl --no-pager
# rpm -qa | grep -i xen libvirt-daemon-xen-2.2.0-2.fc25.x86_64 jaxen-1.1.6-9.fc25.noarch xen-hypervisor-4.7.1-9.fc25.x86_64 xen-4.7.1-9.fc25.x86_64 xen-runtime-4.7.1-9.fc25.x86_64 xen-licenses-4.7.1-9.fc25.x86_64 xen-libs-4.7.1-9.fc25.x86_64 libvirt-daemon-driver-xen-2.2.0-2.fc25.x86_64 # find /etc -name \*libvirt\* /etc/logrotate.d/libvirtd.qemu /etc/logrotate.d/libvirtd.libxl /etc/logrotate.d/libvirtd /etc/sasl2/libvirt.conf /etc/systemd/system/multi-user.target.wants/libvirtd.service /etc/libvirt /etc/libvirt/libvirtd.conf /etc/libvirt/libvirt.conf /etc/libvirt/libvirt-admin.conf /etc/sysconfig/libvirt-guests /etc/sysconfig/libvirtd # rpm -q -V libvirt-daemon (nothing) Still, I can run `systemctl start libvirtd.service` and it starts properly and works. When I do so (run the service manually), the `journalctl --boot -u libvirtd.service` will return something (it returned nothing (-- No entries --) before manual start): -- Logs begin at Thu 2014-12-04 11:52:14 CET, end at Mon 2017-03-27 09:55:43 CEST. -- Mar 27 09:55:42 zyxPad systemd[1]: Starting Virtualization daemon... Mar 27 09:55:42 zyxPad systemd[1]: Started Virtualization daemon. Mar 27 09:55:43 zyxPad dnsmasq[4009]: started, version 2.76 cachesize 150 Mar 27 09:55:43 zyxPad dnsmasq[4009]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify Mar 27 09:55:43 zyxPad dnsmasq-dhcp[4009]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h Mar 27 09:55:43 zyxPad dnsmasq-dhcp[4009]: DHCP, sockets bound exclusively to interface virbr0 Mar 27 09:55:43 zyxPad dnsmasq[4009]: reading /etc/resolv.conf Mar 27 09:55:43 zyxPad dnsmasq[4009]: using nameserver 10.38.5.26#53 Mar 27 09:55:43 zyxPad dnsmasq[4009]: using nameserver 10.35.255.14#53 Mar 27 09:55:43 zyxPad dnsmasq[4009]: using nameserver 10.0.0.138#53 Mar 27 09:55:43 zyxPad dnsmasq[4009]: read /etc/hosts - 5 addresses Mar 27 09:55:43 zyxPad dnsmasq[4009]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses Mar 27 09:55:43 zyxPad dnsmasq-dhcp[4009]: read /var/lib/libvirt/dnsmasq/default.hostsfile Some of the nameserver-s are for VPN, which is currently active. Just in case you might find this useful (even you did not ask for it): # rpm -qa | grep libvirt | sort libvirt-client-2.2.0-2.fc25.x86_64 libvirt-daemon-2.2.0-2.fc25.x86_64 libvirt-daemon-config-network-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-interface-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-libxl-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-network-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-nodedev-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-nwfilter-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-qemu-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-secret-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-storage-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-xen-2.2.0-2.fc25.x86_64 libvirt-daemon-kvm-2.2.0-2.fc25.x86_64 libvirt-daemon-qemu-2.2.0-2.fc25.x86_64 libvirt-daemon-xen-2.2.0-2.fc25.x86_64 libvirt-gconfig-1.0.0-1.fc25.x86_64 libvirt-glib-1.0.0-1.fc25.x86_64 libvirt-gobject-1.0.0-1.fc25.x86_64 libvirt-libs-2.2.0-2.fc25.x86_64 libvirt-python-2.2.0-1.fc25.x86_64
Hmm didn't realize plain systemctl output is incomplete. Can you also provide: systemctl list-units --all --no-pager
Created attachment 1266841 [details] systemctl list-units --all --no-pager
Okay I don't really know what's going or what more to check for. Reassigning to systemd mostly for feedback from maintainers over there systemd folks, user has libvirtd.service enabled, but it doesn't seem to even attempt to start at boot, nothing in journalctl --boot to indicate it that I can see, but when user manually does 'systemctl start libvirtd.service' everything works fine. ideas about what could be causing this/what to check for?
Could someone please escalate this with the systemd people? It happens continuously on my system. I wonder why the libvirtd.service depends on the apparmor.service even though that doesn't exist on a Fedora system?
Sorry for the delay. > I wonder why the libvirtd.service depends on the apparmor.service even though that doesn't exist on a Fedora system? At least one easy question: it doesn't. It is only specified to start after apparmor.service it was installed (and part of the initial transaction). And now the the original issue: after a reboot, before starting libvirtd.service manually, please paste the output of: systemctl show -p State,WantedBy,RequiredBy,After,Before,PartOf,Conflicts,ActiveState,SubState,LoadState libvirtd
> please paste the output of: > ... Property State does not exist. PartOf= RequiredBy= WantedBy=multi-user.target Conflicts=shutdown.target Before=shutdown.target libvirt-guests.service multi-user.target After=iscsid.service remote-fs.target system.slice dbus.service local-fs.target network.target sysinit.target basic.target apparmor.service xenstored.service systemd-journald.socket LoadState=loaded ActiveState=inactive SubState=dead
I agree with Milan Crha - I get the same output: % LC_ALL=C sudo systemctl show -p State,WantedBy,RequiredBy,After,Before,PartOf,Conflicts,ActiveState,SubState,LoadState libvirtd Property State does not exist. PartOf= RequiredBy= WantedBy=multi-user.target Conflicts=shutdown.target Before=libvirt-guests.service multi-user.target shutdown.target After=sysinit.target dbus.service iscsid.service virtlogd.socket system.slice xenstored.service remote-fs.target systemd-journald.socket network.target local-fs.target apparmor.service basic.target virtlogd.service LoadState=loaded ActiveState=inactive SubState=dead
Hm, can one of you please boot with 'systemd.log-level=debug' on the kernel command line and attach the whole 'journalctl -b' output. Thanks.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #20) > Hm, can one of you please boot with 'systemd.log-level=debug' on the kernel > command line I'm sorry, but how do you do that these days, please? Just by editing the boot line in grub? Also, the whole journalctl log contains some information I consider private (like MAC addresses, WiFi networks and so on). How to strip them easily, without wasting an hour on it?
It's probably better to just edit the command line in the grub menu. That only affects a single boot. There's no automatic way to filter stuff afaik.
Created attachment 1292821 [details] journalctl -b I'm not sure if I did it right, I expected longer log, but here you are.
Created attachment 1299447 [details] journalctl -b from jlmagee shortly after boot
Created attachment 1299448 [details] requested libvirtd unit properties from jlmagee
added attachments for items requested earlier in thread This is a fully update F26 with virt-preview. I do not think is has anything to do with apparmor. It is whatever causes the "conflicted_by=yes" below but I haven't figured out a way to debug the cause of that. systemctl status libvirtd.service ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: inactive (dead) Docs: man:libvirtd(8) http://libvirt.org Jul 16 15:06:20 mnetjlm7.mageenet.net systemd[1]: libvirtd.service: Looking at job libvirtd.service/stop conflicted_by=yes Jul 16 15:06:20 mnetjlm7.mageenet.net systemd[1]: libvirtd.service: Looking at job libvirtd.service/start conflicted_by=no Jul 16 15:06:20 mnetjlm7.mageenet.net systemd[1]: libvirtd.service: Fixing conflicting jobs libvirtd.service/stop,libvirtd.service/start by deleting job libvirtd.service/start
zbyszek, please check John's output above when you get a moment
Hmm, so the issue is that some other service has Conflicts=libvirtd.service, and when they are both pulled in, libvirtd.service/start job gets kicked from the transaction. For example, I have a machine with chronyd.service and systemd-timesyncd.service both enabled, and during boot I get: Aug 08 00:14:43 rawhide systemd[1]: Pulling in systemd-timesyncd.service/start from sysinit.target/start Aug 08 00:14:43 rawhide systemd[1]: Pulling in chronyd.service/stop from systemd-timesyncd.service/start Aug 08 00:14:43 rawhide systemd[1]: Pulling in systemd-timesyncd.service/stop from chronyd.service/start Aug 08 00:14:43 rawhide systemd[1]: Pulling in chronyd.service/start from multi-user.target/start Aug 08 00:15:03 rawhide systemd[1]: chronyd.service: chronyd.service/stop would change existing job. Aug 08 00:15:03 rawhide systemd[1]: chronyd.service: Deleting chronyd.service/stop to minimize impact. so it's essentially pseudo-random which service will win. Something similar happens in the case of libvirt, with the unfortunate consequence that libvirtd.service/start gets kicked out. Unfortunately systemd's reporting is really bad (logs above are from patched systemd, and vanilla systemd prints nothing useful.) One of the problems is that even though Conflicts is symmetrical, it is *not* reported in the conflicted unit, only in the one that declares the conflict. So it's not possible to just look at libvirtd.service, it is necessary to guess what is the conflicting unit. I'll take this upstream to improve the debug logs for this. Hah, but I think I found the culprit: /usr/lib/systemd/system/xendomains.service: Conflicts=libvirtd.service OK, so maybe I was overly critical, the answer is right there in the logs: Jul 16 15:06:20 mnetjlm7.mageenet.net systemd[1]: xendomains.service: Looking at job xendomains.service/start conflicted_by=no Jul 16 15:06:20 mnetjlm7.mageenet.net systemd[1]: xendomains.service: Looking at job xendomains.service/stop conflicted_by=no Jul 16 15:06:20 mnetjlm7.mageenet.net systemd[1]: xendomains.service: Fixing conflicting jobs xendomains.service/start,xendomains.service/stop by deleting job xendomains.service/stop Jul 16 15:06:20 mnetjlm7.mageenet.net systemd[1]: libvirtd.service: Looking at job libvirtd.service/stop conflicted_by=yes Jul 16 15:06:20 mnetjlm7.mageenet.net systemd[1]: libvirtd.service: Looking at job libvirtd.service/start conflicted_by=no Jul 16 15:06:20 mnetjlm7.mageenet.net systemd[1]: libvirtd.service: Fixing conflicting jobs libvirtd.service/stop,libvirtd.service/start by deleting job libvirtd.service/start This seems to be a bug in xendomains.service, I don't it has any business in preventing libvirtd.service from running. Tentatively reassiging to xen. -- @xen maintainers: tl;dr version: /usr/lib/systemd/system/xendomains.service has Conflicts=libvirtd.service which prevents libvirtd.service from running in the initial transaction, which makes people unhappy. This most likely affects F25-rawhide.
Thank you zbyszek! Looks like the xen commit that added the Conflicts line is: commit b38a310f9f21d95f98d3cce915787c34cb5f727c Author: Olaf Hering <olaf> Date: Thu Oct 29 11:02:54 2015 +0000 tools/hotplug: xendomains.service conflicts with libvirt xendomains will manage guests behind libvirts back: - libvirt starts a guest - that guest can be "managed" by libvirt and xl at the same time - when xendomains runs on shutdown it will save the guest using xl libvirt does not know about this - when xendomains runs on boot it will restore the saved guest using xl libvirt does not know about this, it will just fail to manage the restored guest To prevent xendomains from interfering with libvirt add a Conflicts= to xendomains.service. It will cause libvirt to be stopped if xendomains is started manually with 'systemctl start'.
Milan/Allan/John can you guys confirm that xendomains service is enabled? If you want to use libvirt tools the equivalent service is libvirt-guests Though it would be nice if systemd gave hints in non-debug 'systemctl status' output that libvirt didn't start on boot due to a conflict
Yup it is enabled - there was nothing in the release notes regarding xen not being able to co-exist on the system. systemctl status xendomains.service ● xendomains.service - Xendomains - start and stop guests on boot and shutdown Loaded: loaded (/usr/lib/systemd/system/xendomains.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at ons 2017-08-09 09:46:48 CEST; 20min ago └─ ConditionPathExists=/proc/xen/capabilities was not met
I can also confirm disabling xendomains will make libvirtd start correctly on reboot. ● xendomains.service - Xendomains - start and stop guests on boot and shutdown Loaded: loaded (/usr/lib/systemd/system/xendomains.service; disabled; vendor preset: enabled) Active: inactive (dead) ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since ons 2017-08-09 11:49:38 CEST; 3min 58s ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 1491 (libvirtd) Tasks: 23 (limit: 4915) Memory: 62.6M CPU: 2.076s CGroup: /system.slice/libvirtd.service ├─1491 /usr/sbin/libvirtd ├─3267 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper └─3268 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
(In reply to Cole Robinson from comment #30) > Milan/Allan/John can you guys confirm that xendomains service is enabled? If > you want to use libvirt tools the equivalent service is libvirt-guests Mine looks similarly as comment #31. Disabling it with: $ systemctl disable xendomains.service and after restart libvirtd service runs again, as expected. I'll keep xendomains.service disabled now.
xen-4.8.1-7.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-b8fa8e1a13
xen-4.8.1-7.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-b8fa8e1a13
xen-4.7.3-3.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-978bebe3a7
xen-4.8.1-7.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
xen-4.7.3-3.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-978bebe3a7
xen-4.7.3-4.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-ed735463e3
xen-4.7.3-4.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-ed735463e3
xen-4.7.3-4.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.