Created attachment 600785 [details] proposed patch to fix the systemd service definition Description of problem: libvirt-guests is configured to gracefully shut down all guest vms on host shutdown: $ cat /etc/sysconfig/libvirt-guests ON_SHUTDOWN=shutdown PARALLEL_SHUTDOWN=10 SHUTDOWN_TIMEOUT=300 But the guests are killed hard on system shutdown. Version-Release number of selected component (if applicable): RPM-Versions: fedora-release-17-1.noarch libvirt-client-0.9.11.4-3.fc17.x86_64 systemd-44-17.fc17.x86_64 How reproducible: Always Additional info: systemd considers libvirt-guests.service to be stopped when the system is running: $ systemctl status libvirt-guests.service libvirt-guests.service - Suspend Active Libvirt Guests Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; enabled) Active: deactivating (stop) since Fri, 27 Jul 2012 15:47:31 +0200; 2min 48s ago Process: 1085 ExecStart=/etc/init.d/libvirt-guests start (code=exited, status=0/SUCCESS) Control: 1150 (libvirt-guests) CGroup: name=systemd:/system/libvirt-guests.service └ control ├ 1150 /bin/sh /etc/init.d/libvirt-guests stop └ 2257 sleep 1 libvirt-guests.service is defined as type "simple" in systemd (the default). That means systemd will shut down the service when the start executable is terminated after starting is done. Systemd will not call stop again on system shutdown because it thinks it is already stopped. The solution is to define it as type "oneshot" and set the flag "RemainAfterExit". Then systemd will consider the service as active after startup and will call the stop function on host shutdown. With the attached patch everything works as expected.
Can you please post this patch upstream to libvir-list? It will get a faster review there.
Patch posted upstream for review, see https://www.redhat.com/archives/libvir-list/2012-July/msg01594.html
Moving to POST - we need to backport the following into the next build for F17: commit 79ca7e4e5763795b041e95738c73d4e64bf1024b Author: Gerd v. Egidy <lists> Date: Fri Jul 27 17:01:04 2012 +0200 libvirt-guests: systemd host shutdown does not work
*** Bug 834592 has been marked as a duplicate of this bug. ***
libvirt-0.9.11.5-1.fc17 has been submitted as an update for Fedora 17. https://admin.fedoraproject.org/updates/libvirt-0.9.11.5-1.fc17
Package libvirt-0.9.11.5-1.fc17: * should fix your issue, * was pushed to the Fedora 17 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing libvirt-0.9.11.5-1.fc17' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-11838/libvirt-0.9.11.5-1.fc17 then log in and leave karma (feedback).
libvirt-0.9.11.5-2.fc17 has been submitted as an update for Fedora 17. https://admin.fedoraproject.org/updates/libvirt-0.9.11.5-2.fc17
Although not trying to gracefully shutdown guests on host shutdown, I believe my problem is related to this bug. Fully updated host from updates + libvirt from updates-testing: $ rpm -q libvirt libvirt-0.9.11.5-2.fc17.x86_64 However any guests running before host reboot still fail to automatically restore. virsh list --all shows them all in a shutdown state. This all worked "out of the box" on F16. For now I am forced to virsh save, host reboot, virsh restore. My /etc/sysconfig/libvirt-guests file only consists of commented lines and I believe that the default action should be to restore previously running guests. I had previously amended /usr/lib/systemd/system/libvirt-guests.service as described in other bug reports: $ cat /usr/lib/systemd/system/libvirt-guests.service [Unit] Description=Suspend Active Libvirt Guests After=syslog.target network.target [Service] EnvironmentFile=-/etc/sysconfig/libvirt-guests # Hack just call traditional service until we factor # out the code ExecStart=/etc/init.d/libvirt-guests start ExecStop=/etc/init.d/libvirt-guests stop Type=oneshot RemainAfterExit=yes [Install] WantedBy=multi-user.target So I now get: $ systemctl status libvirt-guests.service libvirt-guests.service - Suspend Active Libvirt Guests Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; enabled) Active: active (exited) since Thu, 16 Aug 2012 09:41:53 +0100; 1h 50min ago Process: 1085 ExecStart=/etc/init.d/libvirt-guests start (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/libvirt-guests.service I have not yet posted any karma on the new libvirt package in case this is a separate issue. Dave
libvirt-0.9.11.5-3.fc17 has been submitted as an update for Fedora 17. https://admin.fedoraproject.org/updates/libvirt-0.9.11.5-3.fc17
Posted negative karma against the fix, but my comments all came out on one line so re-posting here. # rpm -q libvirt libvirt-0.9.11.5-3.fc17.x86_64 # systemctl status libvirt-guests.service libvirt-guests.service - Suspend Active Libvirt Guests Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; enabled) Active: active (exited) since Thu, 23 Aug 2012 17:34:44 +0100; 8min ago Process: 1093 ExecStart=/etc/init.d/libvirt-guests start (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/libvirt-guests.service However my VMs are still in a shutdown state after host reboot. I have some mathematical simulations that need to run for many days, so I thought a VM would be the best place for them, but I have to: virsh save .... virsh restore... virsh list --all (running) reboot host virsh list --all (showing as shutdown!) virsh restore ... (running) Loved the way this 'just worked' on F16! Dave
Hi Dave, thanks for reporting your issue here. Please help to find out what is going on with guest shutdown on your system: 1. become root: su 2. start all your vms 3. check the service: systemctl status libvirt-guests.service it should be loaded, enabled and active (exited) 4. tell the service to shut the vms down: systemctl stop libvirt-guests.service this should take some time 5. check the service: systemctl status libvirt-guests.service it should be inactive (dead) 6. check your vms: virsh list --all they should now be shutdown 7. start the service again: systemctl start libvirt-guests.service 8. check the service: systemctl status libvirt-guests.service it should be active (exited) again 9. check your vms: virsh list -all they should now be running Please report back here if and where your results differ from the expected.
(In reply to comment #11) > Hi Dave, > > thanks for reporting your issue here. Please help to find out what is going > on with guest shutdown on your system: > > > 6. check your vms: virsh list --all > they should now be shutdown Also useful would be: virsh list --all --managed-save the guests that were previously running should now be listed as managed save, rather than shutdown.
Hi Gerd & Eric, Thanks for your replies. All VMs running: # virsh list --all Id Name State ---------------------------------------------------- 2 centos6 running 3 centos6-clone running # systemctl status libvirt-guests.service libvirt-guests.service - Suspend Active Libvirt Guests Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; enabled) Active: active (exited) since Thu, 23 Aug 2012 17:56:59 +0100; 5h 3min ago Process: 1100 ExecStart=/etc/init.d/libvirt-guests start (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/libvirt-guests.service OK so far! However # systemctl stop libvirt-guests.service returned instantly rather that the delay you mentioned and gave a SELinux error: "SELinux is preventing /usr/bin/virsh from getattr access on the file /usr/sbin/libvirtd." It did stop the service though: # systemctl status libvirt-guests.service libvirt-guests.service - Suspend Active Libvirt Guests Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; enabled) Active: inactive (dead) since Thu, 23 Aug 2012 23:03:03 +0100; 2min 20s ago Process: 8171 ExecStop=/etc/init.d/libvirt-guests stop (code=exited, status=0/SUCCESS) Process: 1100 ExecStart=/etc/init.d/libvirt-guests start (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/libvirt-guests.service But the VMs are still running! # virsh list --all Id Name State ---------------------------------------------------- 2 centos6 running 3 centos6-clone running # virsh list --all --managed-save Id Name State ---------------------------------------------------- 2 centos6 running 3 centos6-clone running Trying again with SELinux in permissive mode because of the above error: # setenforce permissive # systemctl stop libvirt-guests.service # virsh list --all Id Name State ---------------------------------------------------- 2 centos6 running 3 centos6-clone running # systemctl status libvirt-guests.service libvirt-guests.service - Suspend Active Libvirt Guests Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; enabled) Active: inactive (dead) since Thu, 23 Aug 2012 23:08:04 +0100; 4min 21s ago Process: 8279 ExecStop=/etc/init.d/libvirt-guests stop (code=exited, status=0/SUCCESS) Process: 8253 ExecStart=/etc/init.d/libvirt-guests start (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/libvirt-guests.service Don't forget that my specific issue is that a host reboot does not suspend the VMs, but I am posting it here as I believed that it was the job of libvirt-guests.service to do the shutdown or suspend/resume on reboot and that as the original poster's fix to the way libvirt-guests.service was started solved my problem initially for a few days then it must be related. Hope this helps narrow down the issue and thanks again for replying. Dave
(In reply to comment #13) > # systemctl stop libvirt-guests.service > returned instantly rather that the delay you mentioned and gave a SELinux > error: > > "SELinux is preventing /usr/bin/virsh from getattr access on the file > /usr/sbin/libvirtd." Ok, that really looks like a SELinux issue. > It did stop the service though: yeah, but not the way it was intended. > Trying again with SELinux in permissive mode because of the above error: > > # setenforce permissive > # systemctl stop libvirt-guests.service was the service active at this point again? If not, please try it again and make sure with a systemctl status call that it is really active. If SELinux really was the issue, it should work then. > Don't forget that my specific issue is that a host reboot does not suspend > the VMs, but I am posting it here as I believed that it was the job of > libvirt-guests.service to do the shutdown or suspend/resume on reboot and > that as the original poster's fix to the way libvirt-guests.service was > started solved my problem initially for a few days then it must be related. Yes, shutdown & suspend are closely related.
Well I was sure I had the service running at the time I changed the SELinux mode! However: # systemctl status libvirt-guests.service libvirt-guests.service - Suspend Active Libvirt Guests Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; enabled) Active: active (exited) since Thu, 23 Aug 2012 23:18:13 +0100; 1h 6min ago Process: 8279 ExecStop=/etc/init.d/libvirt-guests stop (code=exited, status=0/SUCCESS) Process: 8429 ExecStart=/etc/init.d/libvirt-guests start (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/libvirt-guests.service [root@workstation ~]# setenforce permissive [root@workstation ~]# systemctl stop libvirt-guests.service [root@workstation ~]# systemctl status libvirt-guests.service libvirt-guests.service - Suspend Active Libvirt Guests Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; enabled) Active: inactive (dead) since Fri, 24 Aug 2012 00:24:54 +0100; 5s ago Process: 9608 ExecStop=/etc/init.d/libvirt-guests stop (code=exited, status=0/SUCCESS) Process: 8429 ExecStart=/etc/init.d/libvirt-guests start (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/libvirt-guests.service Aug 24 00:24:49 workstation.home libvirt-guests[9608]: Running guests on default URI: centos6, centos6-clone Aug 24 00:24:49 workstation.home libvirt-guests[9608]: Suspending guests on default URI... Aug 24 00:24:51 workstation.home libvirt-guests[9608]: [79B blob data] Aug 24 00:24:54 workstation.home libvirt-guests[9608]: [137B blob data] [root@workstation ~]# virsh list --all Id Name State ---------------------------------------------------- - centos6 shut off - centos6-clone shut off # virsh list --all --managed-save Id Name State ---------------------------------------------------- - centos6 saved - centos6-clone saved Thats more like it! I now have 4 SELinux alerts: #1 SELinux is preventing /usr/bin/virsh from getattr access on the file /usr/sbin/libvirtd. ***** Plugin catchall (100. confidence) suggests *************************** If you believe that virsh should be allowed getattr access on the libvirtd file by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep virsh /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp Additional Information: Source Context system_u:system_r:virsh_t:s0 Target Context system_u:object_r:virtd_exec_t:s0 Target Objects /usr/sbin/libvirtd [ file ] Source virsh Source Path /usr/bin/virsh Port <Unknown> Host workstation.home Source RPM Packages libvirt-client-0.9.11.5-3.fc17.x86_64 Target RPM Packages libvirt-daemon-0.9.11.5-3.fc17.x86_64 Policy RPM selinux-policy-3.10.0-145.fc17.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Permissive Host Name workstation.home Platform Linux workstation.home 3.5.2-3.fc17.x86_64 #1 SMP Tue Aug 21 19:06:52 UTC 2012 x86_64 x86_64 Alert Count 6 First Seen 2012-08-23 23:03:03 BST Last Seen 2012-08-24 00:28:54 BST Local ID 3973ae67-aed3-4854-afcd-f72fea85862d Raw Audit Messages type=AVC msg=audit(1345764534.210:325): avc: denied { getattr } for pid=9932 comm="virsh" path="/usr/sbin/libvirtd" dev="dm-1" ino=183308 scontext=system_u:system_r:virsh_t:s0 tcontext=system_u:object_r:virtd_exec_t:s0 tclass=file type=SYSCALL msg=audit(1345764534.210:325): arch=x86_64 syscall=stat success=yes exit=0 a0=7f2a7edb6cac a1=7fffbb3b8130 a2=7fffbb3b8130 a3=7fffbb3b7ec0 items=0 ppid=9931 pid=9932 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=virsh exe=/usr/bin/virsh subj=system_u:system_r:virsh_t:s0 key=(null) Hash: virsh,virsh_t,virtd_exec_t,file,getattr audit2allow #============= virsh_t ============== allow virsh_t virtd_exec_t:file getattr; audit2allow -R #============= virsh_t ============== allow virsh_t virtd_exec_t:file getattr; --------------------------------------------------------------------------- #2 SELinux is preventing /usr/bin/virsh from read access on the file /etc/libvirt/libvirt.conf. ***** Plugin catchall (100. confidence) suggests *************************** If you believe that virsh should be allowed read access on the libvirt.conf file by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep virsh /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp Additional Information: Source Context system_u:system_r:virsh_t:s0 Target Context system_u:object_r:virt_etc_t:s0 Target Objects /etc/libvirt/libvirt.conf [ file ] Source virsh Source Path /usr/bin/virsh Port <Unknown> Host workstation.home Source RPM Packages libvirt-client-0.9.11.5-3.fc17.x86_64 Target RPM Packages libvirt-client-0.9.11.5-3.fc17.x86_64 Policy RPM selinux-policy-3.10.0-145.fc17.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Permissive Host Name workstation.home Platform Linux workstation.home 3.5.2-3.fc17.x86_64 #1 SMP Tue Aug 21 19:06:52 UTC 2012 x86_64 x86_64 Alert Count 3 First Seen 2012-08-24 00:24:49 BST Last Seen 2012-08-24 00:28:53 BST Local ID 8ac42743-cb17-4b3c-b5a3-a13f0266448c Raw Audit Messages type=AVC msg=audit(1345764533.244:300): avc: denied { read } for pid=9846 comm="virsh" name="libvirt.conf" dev="dm-1" ino=524642 scontext=system_u:system_r:virsh_t:s0 tcontext=system_u:object_r:virt_etc_t:s0 tclass=file type=AVC msg=audit(1345764533.244:300): avc: denied { open } for pid=9846 comm="virsh" path="/etc/libvirt/libvirt.conf" dev="dm-1" ino=524642 scontext=system_u:system_r:virsh_t:s0 tcontext=system_u:object_r:virt_etc_t:s0 tclass=file type=SYSCALL msg=audit(1345764533.244:300): arch=x86_64 syscall=open success=yes exit=ENXIO a0=122d930 a1=0 a2=7ffff36bd2e8 a3=7ffff36bd060 items=0 ppid=9843 pid=9846 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=virsh exe=/usr/bin/virsh subj=system_u:system_r:virsh_t:s0 key=(null) Hash: virsh,virsh_t,virt_etc_t,file,read audit2allow #============= virsh_t ============== allow virsh_t virt_etc_t:file { read open }; audit2allow -R #============= virsh_t ============== allow virsh_t virt_etc_t:file { read open }; --------------------------------------------------------------------------- #3 SELinux is preventing /usr/bin/virsh from write access on the sock_file libvirt-sock. ***** Plugin catchall (100. confidence) suggests *************************** If you believe that virsh should be allowed write access on the libvirt-sock sock_file by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep virsh /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp Additional Information: Source Context system_u:system_r:virsh_t:s0 Target Context system_u:object_r:virt_var_run_t:s0 Target Objects libvirt-sock [ sock_file ] Source virsh Source Path /usr/bin/virsh Port <Unknown> Host workstation.home Source RPM Packages libvirt-client-0.9.11.5-3.fc17.x86_64 Target RPM Packages Policy RPM selinux-policy-3.10.0-145.fc17.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Permissive Host Name workstation.home Platform Linux workstation.home 3.5.2-3.fc17.x86_64 #1 SMP Tue Aug 21 19:06:52 UTC 2012 x86_64 x86_64 Alert Count 3 First Seen 2012-08-24 00:24:49 BST Last Seen 2012-08-24 00:28:53 BST Local ID a9738e6f-cf91-47dd-ae84-e7db1597e7e9 Raw Audit Messages type=AVC msg=audit(1345764533.244:302): avc: denied { write } for pid=9846 comm="virsh" name="libvirt-sock" dev="tmpfs" ino=21887 scontext=system_u:system_r:virsh_t:s0 tcontext=system_u:object_r:virt_var_run_t:s0 tclass=sock_file type=AVC msg=audit(1345764533.244:302): avc: denied { connectto } for pid=9846 comm="virsh" path="/run/libvirt/libvirt-sock" scontext=system_u:system_r:virsh_t:s0 tcontext=system_u:system_r:virtd_t:s0-s0:c0.c1023 tclass=unix_stream_socket type=SYSCALL msg=audit(1345764533.244:302): arch=x86_64 syscall=connect success=yes exit=0 a0=6 a1=7ffff36bcf40 a2=6e a3=7ffff36bcc00 items=0 ppid=9843 pid=9846 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=virsh exe=/usr/bin/virsh subj=system_u:system_r:virsh_t:s0 key=(null) Hash: virsh,virsh_t,virt_var_run_t,sock_file,write audit2allow #============= virsh_t ============== allow virsh_t virt_var_run_t:sock_file write; allow virsh_t virtd_t:unix_stream_socket connectto; audit2allow -R #============= virsh_t ============== allow virsh_t virt_var_run_t:sock_file write; allow virsh_t virtd_t:unix_stream_socket connectto; --------------------------------------------------------------------------- #4 SELinux is preventing /usr/bin/virsh from connectto access on the unix_stream_socket /run/libvirt/libvirt-sock. ***** Plugin catchall (100. confidence) suggests *************************** If you believe that virsh should be allowed connectto access on the libvirt-sock unix_stream_socket by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep virsh /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp Additional Information: Source Context system_u:system_r:virsh_t:s0 Target Context system_u:system_r:virtd_t:s0-s0:c0.c1023 Target Objects /run/libvirt/libvirt-sock [ unix_stream_socket ] Source virsh Source Path /usr/bin/virsh Port <Unknown> Host workstation.home Source RPM Packages libvirt-client-0.9.11.5-3.fc17.x86_64 Target RPM Packages Policy RPM selinux-policy-3.10.0-145.fc17.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Permissive Host Name workstation.home Platform Linux workstation.home 3.5.2-3.fc17.x86_64 #1 SMP Tue Aug 21 19:06:52 UTC 2012 x86_64 x86_64 Alert Count 1 First Seen 2012-08-24 00:28:54 BST Last Seen 2012-08-24 00:28:54 BST Local ID d4a88473-b5a1-4df9-ac35-8013e2ebc096 Raw Audit Messages type=AVC msg=audit(1345764534.199:324): avc: denied { connectto } for pid=9925 comm="virsh" path="/run/libvirt/libvirt-sock" scontext=system_u:system_r:virsh_t:s0 tcontext=system_u:system_r:virtd_t:s0-s0:c0.c1023 tclass=unix_stream_socket type=SYSCALL msg=audit(1345764534.199:324): arch=x86_64 syscall=connect success=yes exit=0 a0=6 a1=7fff77eb3000 a2=6e a3=7fff77eb2cc0 items=0 ppid=9924 pid=9925 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=virsh exe=/usr/bin/virsh subj=system_u:system_r:virsh_t:s0 key=(null) Hash: virsh,virsh_t,virtd_t,unix_stream_socket,connectto audit2allow #============= virsh_t ============== allow virsh_t virtd_t:unix_stream_socket connectto; audit2allow -R #============= virsh_t ============== allow virsh_t virtd_t:unix_stream_socket connectto; --------------------------------------------------------------------------- Starting the service restored the VMs correctly. I will leave my system as permissive till this is sorted. Do you think this is a policy error or an incorrect file context that only affects me?
(In reply to comment #15) Hi Dave, I was able to reproduce your problem. Seems like indeed the selinux policy does not allow virsh calls executed from systemd services. I have created bug #851860 for this problem. I did not notice this before because I have to admit that I run my system with selinux disabled. I have some hacks in place for some custom stuff which would be very hard to get into a proper selinux policy.
Thanks Gerd, You have described the bug far better than I could have.
libvirt-0.9.11.5-3.fc17 has been pushed to the Fedora 17 stable repository. If problems still persist, please make note of it in this bug report.
The same problem exits in F18. Here here the package has the version: libvirt-0.10.2.3-1.fc18.x86_64 But no selinux deny entry's.