This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 805942 - systemd kills VMs and dnsmasq if libvirtd.service when restarted
systemd kills VMs and dnsmasq if libvirtd.service when restarted
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: systemd (Show other bugs)
17
Unspecified Unspecified
high Severity urgent
: ---
: ---
Assigned To: systemd-maint
Fedora Extras Quality Assurance
AcceptedBlocker https://fedoraproject...
: CommonBugs
Depends On:
Blocks: F17Blocker/F17FinalBlocker
  Show dependency treegraph
 
Reported: 2012-03-22 10:15 EDT by Mark McLoughlin
Modified: 2012-05-08 18:36 EDT (History)
28 users (show)

See Also:
Fixed In Version: systemd-44-6.fc17
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-04-25 01:02:22 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Mark McLoughlin 2012-03-22 10:15:35 EDT
With libvirt-0.9.10-1.fc17.x86_64, if I stop or restart libvirtd the qemu-kvm processes for VMs or the dnsmasq processes for networks get killed

i.e.

 $> systemd-cgls /system/libvirtd.service
 /system/libvirtd.service:
 ├ 1274 libvirtd --daemon
 ├ 1358 /usr/sbin/dnsmasq ...
 └ 5989 /usr/bin/qemu-kvm ...
 $> systemctl stop /system/libvirtd.service
 $> systemd-cgls /system/libvirtd.service
 (nothing)

Dan's suggestion to use KillMode=process fixes this:

  http://www.redhat.com/archives/libvir-list/2012-March/msg00946.html

However, it doesn't fix restart - it appears systemd is killing off all existing processes in the cgroup:

  http://cgit.freedesktop.org/systemd/systemd/tree/src/service.c?id=75c8e3cf#n2093
Comment 2 Laine Stump 2012-04-08 22:04:37 EDT
I'm experiencing the same behavior with:

libvirt-0.9.11-1.fc17.x86_64
systemd-44-4.fc17.x86_64
systemd-44-4.fc17.i686
libcgroup-0.38-1.fc17.x86_64

That is, "systemctl stop libvirtd.service" does not kill other processes started by libvirtd, but a subsequent "systemctl start libvirtd.service does kill all the processes started by the previous libvirtd (i.e. all instances of dnsmasq, radvd, and (most importantly) qemu-kvm. Likewise, "systemctl restart libvirtd.service" also kills all processes started by the previous libvirtd.

I'm guessing this isn't a problem with libvirtd, but it certainly causes a major regression in libvirtd behavior, one that really *must* be eliminated before F17 is released (it would be very good to get it fixed before F17 beta, but I don't know if it qualifies as a blocker).

Should this be reassigned to systemd?
Comment 3 Jim Meyering 2012-04-10 04:58:11 EDT
I learned the hard way that this bug makes a libvirtd restart
break networking in all guests: http://bugzilla.redhat.com/802475#c29
Comment 4 Daniel Berrange 2012-04-10 14:52:20 EDT
I've done more tests and confirmed this is a change in systemd behaviour between Fedora 16 and 17

On Fedora 16

# virsh start vm1
# virsh list
 Id    Name                           State
----------------------------------------------------
 1     vm1                            running

# systemctl status libvirtd.service
libvirtd.service - Virtualization daemon
	  Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled)
	  Active: active (running) since Tue, 10 Apr 2012 19:47:13 +0100; 1min 17s ago
	Main PID: 24351 (libvirtd)
	  CGroup: name=systemd:/system/libvirtd.service
		  ├ 24301 /usr/libexec/qemu-kvm -S -M pc-0.13 -enable-kvm -m 215 -smp 4,sockets=4,cores=1,threads=1 -name vm1 -uuid c7a3edbd-ed...
		  └ 24351 /usr/sbin/libvirtd

# systemctl stop libvirtd.service
# pgrep qemu-kvm
24301
# systemctl start libvirtd.service
# pgrep qemu-kvm
24301
# virsh list
 Id    Name                           State
----------------------------------------------------
 1     vm1                            running

# systemctl status libvirtd.service
libvirtd.service - Virtualization daemon
	  Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled)
	  Active: active (running) since Tue, 10 Apr 2012 19:50:16 +0100; 1s ago
	Main PID: 24655 (libvirtd)
	  CGroup: name=systemd:/system/libvirtd.service
		  ├ 24301 /usr/libexec/qemu-kvm -S -M pc-0.13 -enable-kvm -m 215 -smp 4,sockets=4,cores=1,threads=1 -name vm1 -uuid c7a3edbd-ed...
		  └ 24655 /usr/sbin/libvirtd


Notice libvirtd PID changed, but QEMU stayed the same.


Repeat the same test scenario on Fedora 17 and the QEMU pid is killed off when 'systemctl start libvirtd.service' is run


The libvirtd.service file is

# cat /lib/systemd/system/libvirtd.service 
# NB we don't use socket activation. When libvirtd starts it will
# spawn any virtual machines registered for autostart. We want this
# to occur on every boot, regardless of whether any client connects
# to a socket. Thus socket activation doesn't have any benefit

[Unit]
Description=Virtualization daemon
After=syslog.target
After=udev.target
After=avahi.target
After=dbus.target
Before=libvirt-guests.service

[Service]
KillMode=process
EnvironmentFile=-/etc/sysconfig/libvirtd
ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
ExecReload=/bin/kill -HUP $MAINPID
# Override the maximum number of opened files
#LimitNOFILE=2048

[Install]
WantedBy=multi-user.target



Note how we explicitly use KillMode=process to tell systemd to leave all our VM processes alone when shutting down libvirtd.
Comment 5 Daniel Berrange 2012-04-10 14:53:52 EDT
Browsing systemd GIT logs this looks like a possible culprit

commit 8f53a7b8ea9ba505f8fefe4df4aaa5a8aab1e2eb
Author: Lennart Poettering <lennart@poettering.net>
Date:   Wed Jan 11 01:51:32 2012 +0100

    service: brutally slaughter processes that are running in the cgroup when we enter START_PRE and START



IIUC, it will slaughter all existing processes in the cgroup whenever starting a service..... bye bye VM processes :-(  We really need systemd not todo this if we have KillMode=process
Comment 6 Cole Robinson 2012-04-10 15:09:53 EDT
Proposing as a beta blocker, might just be an F17Blocker though
Comment 7 Adam Williamson 2012-04-10 15:26:17 EDT
-1 blocker, beta or final. It's annoying, but it's a host side issue and can be fixed fine with an update. It doesn't actually prevent virt from working, only causes nastiness if you restart libvirtd. Guest side issues are more likely to merit blocker status, as they will affect live images forever (can't fix lives with an update).



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 8 Eric Blake 2012-04-10 15:31:01 EDT
Is gnome boxes going to be part of a live image?  If that's the case, then you have a host-side issue where restarting libvirtd in a live image host with boxes installed will kill all the VMs running under boxes; at which point, this beta criteria sounds reasonable:
14. The release must be able host virtual guest instances of the same release, using Fedora's current preferred virtualization technology
Comment 9 Jóhann B. Guðmundsson 2012-04-11 05:00:30 EDT
Agreeing with Adam here -1 to blocker beta or final...
Comment 10 Daniel Berrange 2012-04-11 05:23:53 EDT
> -1 blocker, beta or final. It's annoying, but it's a host side issue and can be
> fixed fine with an update. It doesn't actually prevent virt from working, only
> causes nastiness if you restart libvirtd.

libvirtd is restarted in %post when doing an RPM update, which IMHO makes this a serious issue worthy of urgent attention. Users will come after us with flaming torches & pitchforks when we kill their VMs off while doing an RPM update post release.
Comment 11 Eric Blake 2012-04-11 10:09:43 EDT
+1 blocker - if running 'yum update' kills your VMs, then this is _not_ something that can be fixed by an update, but must be fixed prior to the release.
Comment 12 Daniel Veillard 2012-04-11 10:59:01 EDT
Agreed we really need to fix this before the release, to me too it's a blocker !

Daniel
Comment 13 Laine Stump 2012-04-11 11:03:48 EDT
> libvirtd is restarted in %post when doing an RPM update, which IMHO makes this
> a serious issue worthy of urgent attention. Users will come after us with
> flaming torches & pitchforks when we kill their VMs off while doing an RPM
> update post release.

This. Assuming that nobody will happen to restart libvirt during normal operations (which is a very big assumption, since it has been common knowledge for a very long time that restarting libvirt will have *no* effect on running guests and as a matter of fact this is part of standard troubleshooting behavior when guest networking doesn't function, for example), the bigger problem with guests won't happen on the front end when F17 is installed and first used, but at some random time in the future when an update is made to libvirt (likely as a part of a larger update, so users may not even notice that libvirt is being updated), but the result will be effectively crashing all running guests on the system.

Does anyone really think it's acceptable to ship Fedora with a set of packages that *will* lead to guests being unceremoniously killed with *no warning* by (for example) a routine security update?

Several libvirt developers just discussed this on a phone call, and we're all in agreement that this problem should qualify as a beta blocker.
Comment 14 Jared Smith 2012-04-11 13:41:27 EDT
I'm voting for +1 blocker, simply from the standpoint of "element of least surprise".  This really should be fixed by final, even if folks aren't comfortable blocking the beta for a fix.
Comment 15 Adam Williamson 2012-04-11 13:45:46 EDT
It would have been nice to have this information _two days ago_ when we could have done something about it, rather than twelve hours before the go/no-go meeting.

If we take this as a Beta blocker now we slip the release another week, putting us three weeks behind schedule. It really would have been nice to hear about this at any time in the last damn three weeks, so we could have actually tried to fix it.

I'm still on the fence about the issue, frankly. No-one should be running any important VMs using a Beta release as the host (says the guy whose web server is currently running on an F17 host...). And you _can_ run VMs on an F17 host, still. They might explode when you do an update, but they run.

I'm definitely +1 final blocker, people should be able to install a final release and safely deploy critical VMs on it. But I'm not sure about Beta blocker.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 16 Daniel Berrange 2012-04-11 13:49:14 EDT
I don't think this needs to be a Beta blocker - AFAIK, there's never any guarantee that upgrades from Beta -> GA are free from surprises.  Final blocker is fine.
Comment 17 Adam Williamson 2012-04-11 13:49:14 EDT
Eric: Boxes is not in the Beta images.
Comment 18 Tim Flink 2012-04-11 13:54:54 EDT
What is the real difference between blocking for this and releasing it as a 0day update?

If someone is already running an installed version of F17, they'll hit this whether we block for it or not.

The only way that someone would hit this if they aren't going to hit it already is if they install libvirt as part of their installation and start relying on it before updating the system post-install.

My assumption is that users of a pre-release are more likely to do one of the following:
 - Update after install and before doing much with the installed system
 - Install libvirt post-install (which would grab the pkg from updates, not the
   install repo)
 - Use updates-testing as a source repo during installation

If I'm understanding the issue correctly, any of those three scenarios would avoid hitting this bug.

Unless I'm way off on my understanding or my assumptions, I'm -1 beta blocker on this. The risk of causing issues does not seem to increase much if we release beta with the old libvirt.

Final blocker, maybe but we can cross that bridge if/when we get there
Comment 19 Robyn Bergeron 2012-04-11 14:00:56 EDT
I'm with Adam on +1 final -- Beta I am having a hard time seeing (pitchforks
and torches regardless, since we apparently shipped Alpha this way and save for
this bug, haven't seen a lot of pitchforks and torches).
Comment 20 Adam Williamson 2012-04-11 14:02:59 EDT
tflink: you are somewhat off, because the bug appears to be in systemd not libvirt. Though if we get a 0-day fix out for systemd, it should certainly mitigate the issue.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 21 Tim Flink 2012-04-11 14:33:14 EDT
(In reply to comment #20)
> tflink: you are somewhat off, because the bug appears to be in systemd not
> libvirt. Though if we get a 0-day fix out for systemd, it should certainly
> mitigate the issue.

Well, that gets rid of one of my scenarios for not hitting this.

I think that I'm still -1 beta blocker, +1 beta NTH and +1 final blocker. This is a conditional violation of the release criteria (if you're running VMs when you do a system update on the host) and I think that it's enough of a corner/unwise case for a pre-release to justify passing on it as a blocker for beta.
Comment 22 Tom "spot" Callaway 2012-04-11 14:34:56 EDT
I think I have a straightforward fix for this in systemd, but I will need people to test it. Doing a scratch build now.
Comment 23 Tom "spot" Callaway 2012-04-11 14:43:27 EDT
Scratch Build: http://koji.fedoraproject.org/koji/taskinfo?taskID=3980250

If this resolves the issue, I'll take it to the systemd upstream to see if they will take it.
Comment 24 Dennis Gilmore 2012-04-11 15:38:59 EDT
im -1 on beta blocker here, but i do think its a final blocker
Comment 25 Laine Stump 2012-04-11 15:44:26 EDT
I just tested spot's scratch build of systemd and it definitely fixes the problem - dnsmasq and qemu-kvm processes now survive a restart of libvirtd.service. So, along with all the other + and - karma here, a +1 for spot's patch :-)
Comment 26 Tom "spot" Callaway 2012-04-11 16:13:41 EDT
I spoke to Kay about this patch, and he has a different approach to solving this bug. 

<kay> might get fixed tonight or tomorrow
<kay> depends how it will end up, but we have an idea
<kay> end up and how much work it will be, i meant
<kay> we must enforce that kind of killing, always. current plan is to put all ExecPre/Post in a sub-cgroup and kill all them, and only them, when leaving the Pre/Post transaction
<kay> Pre and Post have been misused to start things, we can not track anymore. we kind of must prevent that
<kay> it will get out of control otherwise
<kay> and we promised to provided babysitting data :)
<kay> but we want to be strict here, because we've seen crazy service files we do not want to support in the end

Given kay's timetable, I think this should be resolved well before final, so I'm fine with this being a final blocker and not a beta blocker.
Comment 27 Adam Williamson 2012-04-11 16:14:19 EDT
we seem to have a reasonable consensus on -1 beta blocker here - -1 votes from me and Tim (QA), Dennis (releng), and Daniel Berrange (virt devel). so removing F17Beta. We seem to be solidly +1 final blocker, so marking as acceptedblocker for final.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 29 Daniel Berrange 2012-04-19 04:42:49 EDT
Can we get that fix backported to Fedora 17, or will F17 be rebased to a new systemd release before GA ?  We're ready to test the behaviour with libvirt as soon as new Fedora packages are available.
Comment 30 Michal Schmidt 2012-04-19 05:05:44 EDT
There will be a backport.
Comment 31 Fedora Update System 2012-04-23 20:52:16 EDT
systemd-44-6.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/systemd-44-6.fc17
Comment 32 Fedora Update System 2012-04-23 23:16:31 EDT
Package systemd-44-6.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-44-6.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-6456/systemd-44-6.fc17
then log in and leave karma (feedback).
Comment 33 Fedora Update System 2012-04-25 01:02:22 EDT
systemd-44-6.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 34 Michal Schmidt 2012-05-02 18:12:00 EDT
I had to revert the fix in 44-8.fc17 due to bug 816842. But I also disabled the killing of leftover processes when entering START_PRE or START states, so it should be fine.
Please test if this bug is still fixed in systemd-44-8.fc17.
Comment 35 Adam Williamson 2012-05-08 18:36:33 EDT
Seems to be fine, running VMs keep running, with 44-8.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Note You need to log in before you can comment on or make changes to this bug.