Bug 1046469

If you try this with the latest code does it still happen.  Killing a container should stop all processes.

Closing as fixed in current release.  Reopen if it still happens.

Still see it in RHEL7.

To reproduce, you need to start two Fedora based container in privileged mode. If you start only one in privileged mode, there will be no problem.

Dockerfile

FROM centos:centos7
RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs
CMD ["/usr/sbin/init"]


Script

docker build -t test .
docker run -d --privileged test
docker run -d --privileged test

> Closing as fixed in current release.  Reopen if it still happens.

Re-opening.  It's still happening with a RHEL7.2 host, a RHEL7.2 container and a docker-1.8.2-2.el7.  The Dockerfile contains systemd, if that matters.

Steps to reproduce:
1) Create two containers using:
   docker run -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro --privileged test /sbin/init
   docker run -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro --privileged test /sbin/init

2) Start two containers:
   docker exec -it <container-1-id> bash
   docker exec -it <container-2-id> bash

3) Run top on the bare metal host.  You should see an agetty process burning up one cpu:

  PID   USER  PR  NI    VIRT  RES  SHR  S  %CPU %MEM  TIME+    COMMAND
73529   root  20   0  110004  680  660  R  99.7  0.0  10:53.98  agetty 

Note, this problem does not occur if, when creating the containers, I replace the "--privileged" flag with "--security-opt label:disable --cap-add SYS_ADMIN"

Joe

Not sure this can be fixed.  systemd would have to figure out it is running in a container and then do something different.

Potentially being caused by multiple udevs running?

I have some more tracing and debug available, but...the agetty process is just spinning in the read syscall:

# ps aux|egrep 'USER|19567'|grep -v grep
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     19567 96.8  0.0 110004   796 ?        Rs   15:37  11:08 /sbin/agetty --noclear tty1

# strace -c -p 19567
Process 19567 attached
^CProcess 19567 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.080810           0    168606           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.080810                168606           total

Is read failing?

No

...
read(0, "", 1)                          = 0
read(0, "", 1)                          = 0
read(0, "", 1)                          = 0
read(0, "", 1)                          = 0
...

Here's a little more info, but likely doesn't help.  (It didn't help me).
But I share it anyways.

The agetty process continuously loops through calls from main (agetty.c:372) to get_logname().

(gdb) bac
#0  get_logname (cp=<synthetic pointer>, tp=0x7fff062f3520, op=0x7fff062f3870) at term-utils/agetty.c:1553
#1  main (argc=<optimized out>, argv=<optimized out>) at term-utils/agetty.c:372

The line read at line 1533 is:
Breakpoint 1, get_logname (cp=<synthetic pointer>, tp=0x7fff062f3520, op=0x7fff062f3870) at term-utils/agetty.c:1533
1533				if (read(STDIN_FILENO, &c, 1) < 1) {

After it completes, errno is not set, and the value stored into the variable "c" is:
(gdb) p c
$15 = 3 '\003'

After a bunch of checks, it gets down to the switch stmt at line 1574:
1573				/* Do erase, kill and end-of-line processing. */
1574				switch (key) {

Unfortunately the value of "key" is optimized away.

It falls through to the "default:" case at line 1604, and then executes line 1605:
1602				case CTL('D'):
1603					exit(EXIT_SUCCESS);
1604				default:
1605					if (!isascii(ascval) || !isprint(ascval))
1606						break;

On the call to isprint(), we get to the read() of 1 byte from __fd=0:
(gdb) bac
#0  0x0000000000401fd0 in read@plt ()
#1  0x0000000000403752 in read (__nbytes=1, __buf=0x7fff062f3510, __fd=0) at /usr/include/bits/unistd.h:44
#2  get_logname (cp=<synthetic pointer>, tp=0x7fff062f3520, op=0x7fff062f3870) at term-utils/agetty.c:1533
#3  main (argc=<optimized out>, argv=<optimized out>) at term-utils/agetty.c:372

Unfortunately, even though I'm stepping by instruction, in the optimized binary, I can't stop between the isprint() and the upper level read() at line 1533 (where we started above).

Joe

One more note, wrt to the earlier comment about multiple udevs running:

During the moment that the two privileged containers are started, the number of systemd-udevd processes jumps from 2 to 146.  After about 5 seconds that number drops to 4.

If the same two container invocations are done again, but this time without --privileged, then only 15 systemd-udevd processes are created (again for about 4-5 seconds).

Here's one workaround for this.  Sharing what I've learned.

Adding the following to the docker file 

   rm -f /lib/systemd/system/systemd*udev* ; \
   rm -f /lib/systemd/system/getty.target;

causes both the runnaway agetty and the spike in systemd-udevd processes to go away.  I understand there is no need for udev or getty in containers.


I did also try:
   RUN systemctl disable getty.target
   RUN systemctl disable systemd-udevd.service

and although the Dockerfile built fine, all the getty and udev services were still running as they were previously.  So those two "systemctl disable" appear to be no-ops.  Perhaps they get run before /sbin/init (systemd) is invoked.

This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.