Bug 1018730 - Add workaround for systemd failing to remove cgroups on VM shutdown
Add workaround for systemd failing to remove cgroups on VM shutdown
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Daniel Berrange
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-14 06:17 EDT by Daniel Berrange
Modified: 2014-06-17 20:57 EDT (History)
8 users (show)

See Also:
Fixed In Version: libvirt-1.1.1-9.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-06-13 05:31:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Daniel Berrange 2013-10-14 06:17:50 EDT
Description of problem:
Per

  https://bugzilla.redhat.com/show_bug.cgi?id=988883

systemd-machined will often fail to remove cgroups during VM shutdown. This will then prevent libvirtd starting the VM again until the host is rebooted. This problem appears to affect LXC guests quite alot more than KVM guests, for reasons unknown.

It will become a big problem for RHEL-7 when LXC error reporting is fixed per this bug

  https://bugzilla.redhat.com/show_bug.cgi?id=927072 

The workaround is upstream already

commit bd773e74f0d1d1b9ebbfcaa645178316b4f2265c
Author: Cédric Bosdonnat <cbosdonnat@suse.com>
Date:   Mon Sep 30 16:46:29 2013 +0200

    LXC: workaround machined uncleaned data with containers running systemd.
    
    The problem is described by [0] but its effect on libvirt is that
    starting a container with a full distro running systemd after having
    stopped it simply fails.
    
    The container cleanup now calls the machined Terminate function to make
    sure that everything is in order for the next run.
    
     [0]: https://bugs.freedesktop.org/show_bug.cgi?id=68370


Version-Release number of selected component (if applicable):
libvirt-1.1.1-8.el7

How reproducible:
Random

Steps to Reproduce:
1. virsh -c lxc:/// start guest
2. virsh -c lxc:/// destroy guest
3. goto step 1

Actual results:
Eventually 'start' will fail with an error about cgroups (if you are lucky).

Expected results:


Additional info:
Comment 3 Luwen Su 2013-10-21 01:56:44 EDT
Hi Daniel , 

I still can't meet the error after downgrade the libvirt , systemd , kernel to previous version.

libvirt-1.1.1-8.el7.x86_64
kernel-3.10.0-0.rc7.64.el7.x86_64
systemd-206-1.el7.x86_64

With the command , run 30 minutes 
# while true ; do virsh -c lxc:/// start test ; virsh -c lxc:/// destroy test ; done


Does any other packages related to this issue else ?
Or is there an effective way to reproduce the cgroup error with libvirt?
Thanks .
Comment 4 Daniel Berrange 2013-12-13 07:13:53 EST
(In reply to time.su from comment #3)
> Hi Daniel , 
> 
> I still can't meet the error after downgrade the libvirt , systemd , kernel
> to previous version.
> 
> libvirt-1.1.1-8.el7.x86_64
> kernel-3.10.0-0.rc7.64.el7.x86_64
> systemd-206-1.el7.x86_64
> 
> With the command , run 30 minutes 
> # while true ; do virsh -c lxc:/// start test ; virsh -c lxc:/// destroy
> test ; done
> 
> 
> Does any other packages related to this issue else ?
> Or is there an effective way to reproduce the cgroup error with libvirt?
> Thanks .

It turns out that the key thing is that you want to have systemd itself running inside the container. The best way to get this setup is to use libvirt-sandbox eg

  virt-sandbox-service create -u httpd.service  myapache

and then do your 'while true' loop using the 'myapache' guest.
Comment 5 Luwen Su 2013-12-23 00:28:49 EST
Thanks your kindly help , Daniel.
Verify this bug with libvirt-1.1.1-16.el7.x86_64

Steps:
#virt-sandbox-service create -u httpd.service  myapache
#while true ; do virsh -c lxc:/// start myapache ; virsh -c lxc:/// destroy myapache ; done

Expected:the container can be started and destroyed successfully 


In libvirt-1.1.1-8.el7.x86_64
The steps will cause
1.systemd crashed and generate core dump by abrtd
2.Libvirtd shows
Assertion 'n_pids > 0' failed at src/shared/cgroup-show.c:47, function show_pid_array(). Aborting.
Aborted (core dumped)

3.The container fail to start again due systemd has gone away.
Comment 6 Ludek Smid 2014-06-13 05:31:56 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.