Bug 1202859 - systemd is cleaning up processes from "external" cgroups
Summary: systemd is cleaning up processes from "external" cgroups
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: lxc
Version: epel7
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Thomas Moschny
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-03-17 15:07 UTC by Alexey Kurnosov
Modified: 2018-02-25 13:04 UTC (History)
7 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2018-02-25 13:04:31 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Alexey Kurnosov 2015-03-17 15:07:10 UTC
Description of problem:
In some circumstances systemd moves all tasks from it`s own cgroups 

Version-Release number of selected component (if applicable):
Scientific Linux release 7.1 rolling (Nitrogen)

How reproducible:
Always 

Steps to Reproduce:
1. Login over ssh as a root (without intermediate users).
2. 
[root@v01 ~]# date
Tue Mar 17 17:34:25 MSK 2015
[root@v01 ~]# bash
[root@v01 ~]# for d in  /sys/fs/cgroup/memory; do mkdir "$d/test-1"; echo $$ > "$d/test-1/tasks"; done
[root@v01 ~]# while wc -l   /sys/fs/cgroup/memory/test-1/tasks | grep -qv '^0'; do sleep 1; done; date
3. Open another console
4.
[root@v01 ~]# date
Tue Mar 17 17:34:45 MSK 2015
[root@v01 ~]# echo '*/1 * * * * root /usr/bin/echo 1 > /dev/null' > /etc/cron.d/test
[root@v01 ~]# systemctl enable atd.service #does not matter what service 
[root@v01 ~]# date
Tue Mar 17 17:34:55 MSK 2015

Actual results:
on the first console the script stoped:
Tue Mar 17 17:35:02 MSK 2015
[root@v01 ~]# grep $$ /sys/fs/cgroup/memory/tasks
4901

Expected results:
On the first console the script is still running

Additional info:
If process runs from a derived root session and moved to a some cgroup, whenever systemd is received "enable service" event and if there is a cron event scheduled, when a cron task runs systemd moves all tasks from "external" cgroups to the root cgroup. That is an unacceptable behavior (i lost couple work days for debuging).
P.S. the OS is not actually RHEL, but i believe the bug is applied to RHEL as well.

Comment 2 Alexey Kurnosov 2015-03-17 17:34:52 UTC
systemd version: systemd-208-20.el7.x86_64
P.P.S. the bug randomly occurrences with other work flow, described above way just one 100% reproduced.

Comment 3 Michal Sekletar 2015-03-18 12:28:35 UTC
In RHEL7 cgroup tree in memory controller is managed by systemd. You should not expect that manual changes you make will be preserved. In justified cases there maybe more cgroup managers, but they have cooperate with systemd. For example libvirtd. Also see documentation for option Delegate in systemd.resource-control (5).

Nothing to fix here -> NOTABUG.

Comment 4 Alexey Kurnosov 2015-03-18 13:31:54 UTC
I hit the bug during setup an LXC container (lxc-1.0.7-1.el7.x86_64 from official EPEL repo). So are you saying LXC is incompatible with systemd?

Comment 5 Lukáš Nykrýn 2015-03-18 13:56:01 UTC
It seems that lxc does not register its containers in machined, which means that they will not use Delegate stuff.
http://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers/

By the way why are you using lxc? If I am not mistaken we are supporting libvirt lxc.

Or you can use systemd-nspawn :-)

Comment 6 Alexey Kurnosov 2015-03-19 14:37:16 UTC
>It seems that lxc does not register its containers in machined, which means that they will not use Delegate stuff.

Judging by github commit history "Delegate stuff" added in LXC 1.1. Looks like without this feature LXC is completely useless on 7.x systems. Do you have plans to include LXC 1.1 in EPEL?

>By the way why are you using lxc? If I am not mistaken we are supporting libvirt lxc.
When i looked on libvirt it was a some way outdated (Lack of capability setting, impossible tune many important settings on a container). In general LXC is much more flexible.

Comment 7 Thomas Moschny 2015-04-26 07:41:00 UTC
(In reply to Alexey Kurnosov from comment #6)
> >It seems that lxc does not register its containers in machined, which means that they will not use Delegate stuff.
> 
> Judging by github commit history "Delegate stuff" added in LXC 1.1. Looks
> like without this feature LXC is completely useless on 7.x systems.

Are you speaking of this commit? https://github.com/lxc/lxc/pull/382/commits
If yes, it should be easy to try it out.

> Do you have plans to include LXC 1.1 in EPEL?

Not really. Upstream website says:

  For production environment, try to stick to LXC 1.0.x as this is the long
  term, stable release which we will support until April 2019.

Comment 8 Thomas Moschny 2015-04-26 07:44:18 UTC
Forgot to add this:

For testing purposes, I have lxc 1.1 RPMs here:

http://copr.fedoraproject.org/coprs/thm/lxc1.1/

Comment 9 Alexey Kurnosov 2015-04-26 18:09:12 UTC
>Are you speaking of this commit? https://github.com/lxc/lxc/pull/382/commits
Yes

>If yes, it should be easy to try it out.
I fixed it for me by creating lxc@.service with Delegate=yes, enabling/starting/stopping containers from systemctl and put it in my local repo.

> For production environment, try to stick to LXC 1.0.x as this is the long
>  term, stable release which we will support until April 2019.
It is very strange. Stick to a package which absolutely does not work out of box. At least they should provide a patch to fix it (and include in the EPEL builds).

Comment 10 Paul Szabo 2015-10-26 02:25:32 UTC
I wonder if the Debian bug
http://bugs.debian.org/803013
is related, and whether that patch proposed there
https://bugs.debian.org/cgi-bin/bugreport.cgi?filename=PSz.patch;att=1;msg=5;bug=803013
would solve this issue.

Cheers, Paul

Paul Szabo   psz.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of Sydney    Australia

Comment 11 Fedora Update System 2017-06-10 15:07:55 UTC
lxc-1.0.10-2.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-4aae1e22f1

Comment 12 Fedora Update System 2017-06-11 23:54:33 UTC
lxc-1.0.10-2.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-4aae1e22f1

Comment 13 Fedora Update System 2017-07-30 19:50:37 UTC
lxc-1.0.10-2.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.