Bug 1018730

Summary:	Add workaround for systemd failing to remove cgroups on VM shutdown
Product:	Red Hat Enterprise Linux 7	Reporter:	Daniel Berrangé <berrange>
Component:	libvirt	Assignee:	Daniel Berrangé <berrange>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.0	CC:	acathrow, ajia, berrange, dallan, dyuan, lcui, lsu, svenkatr
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-1.1.1-9.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-06-13 09:31:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Daniel Berrangé 2013-10-14 10:17:50 UTC

Description of problem:
Per

  https://bugzilla.redhat.com/show_bug.cgi?id=988883

systemd-machined will often fail to remove cgroups during VM shutdown. This will then prevent libvirtd starting the VM again until the host is rebooted. This problem appears to affect LXC guests quite alot more than KVM guests, for reasons unknown.

It will become a big problem for RHEL-7 when LXC error reporting is fixed per this bug

  https://bugzilla.redhat.com/show_bug.cgi?id=927072 

The workaround is upstream already

commit bd773e74f0d1d1b9ebbfcaa645178316b4f2265c
Author: Cédric Bosdonnat <cbosdonnat>
Date:   Mon Sep 30 16:46:29 2013 +0200

    LXC: workaround machined uncleaned data with containers running systemd.
    
    The problem is described by [0] but its effect on libvirt is that
    starting a container with a full distro running systemd after having
    stopped it simply fails.
    
    The container cleanup now calls the machined Terminate function to make
    sure that everything is in order for the next run.
    
     [0]: https://bugs.freedesktop.org/show_bug.cgi?id=68370


Version-Release number of selected component (if applicable):
libvirt-1.1.1-8.el7

How reproducible:
Random

Steps to Reproduce:
1. virsh -c lxc:/// start guest
2. virsh -c lxc:/// destroy guest
3. goto step 1

Actual results:
Eventually 'start' will fail with an error about cgroups (if you are lucky).

Expected results:


Additional info:

Comment 3 Luwen Su 2013-10-21 05:56:44 UTC

Hi Daniel , 

I still can't meet the error after downgrade the libvirt , systemd , kernel to previous version.

libvirt-1.1.1-8.el7.x86_64
kernel-3.10.0-0.rc7.64.el7.x86_64
systemd-206-1.el7.x86_64

With the command , run 30 minutes 
# while true ; do virsh -c lxc:/// start test ; virsh -c lxc:/// destroy test ; done


Does any other packages related to this issue else ?
Or is there an effective way to reproduce the cgroup error with libvirt?
Thanks .

Comment 4 Daniel Berrangé 2013-12-13 12:13:53 UTC

(In reply to time.su from comment #3)
> Hi Daniel , 
> 
> I still can't meet the error after downgrade the libvirt , systemd , kernel
> to previous version.
> 
> libvirt-1.1.1-8.el7.x86_64
> kernel-3.10.0-0.rc7.64.el7.x86_64
> systemd-206-1.el7.x86_64
> 
> With the command , run 30 minutes 
> # while true ; do virsh -c lxc:/// start test ; virsh -c lxc:/// destroy
> test ; done
> 
> 
> Does any other packages related to this issue else ?
> Or is there an effective way to reproduce the cgroup error with libvirt?
> Thanks .

It turns out that the key thing is that you want to have systemd itself running inside the container. The best way to get this setup is to use libvirt-sandbox eg

  virt-sandbox-service create -u httpd.service  myapache

and then do your 'while true' loop using the 'myapache' guest.

Comment 5 Luwen Su 2013-12-23 05:28:49 UTC

Thanks your kindly help , Daniel.
Verify this bug with libvirt-1.1.1-16.el7.x86_64

Steps:
#virt-sandbox-service create -u httpd.service  myapache
#while true ; do virsh -c lxc:/// start myapache ; virsh -c lxc:/// destroy myapache ; done

Expected:the container can be started and destroyed successfully 


In libvirt-1.1.1-8.el7.x86_64
The steps will cause
1.systemd crashed and generate core dump by abrtd
2.Libvirtd shows
Assertion 'n_pids > 0' failed at src/shared/cgroup-show.c:47, function show_pid_array(). Aborting.
Aborted (core dumped)

3.The container fail to start again due systemd has gone away.

Comment 6 Ludek Smid 2014-06-13 09:31:56 UTC

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.