Bug 1283891 - When initscripts is installed, systemd in Docker container shows error during systemctl daemon-reload
When initscripts is installed, systemd in Docker container shows error during...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker (Show other bugs)
7.2
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Daniel Walsh
atomic-bugs@redhat.com
: Extras
Depends On:
Blocks: 1283777
  Show dependency treegraph
 
Reported: 2015-11-20 03:03 EST by Jan Pazdziora
Modified: 2016-06-23 12:17 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-06-23 12:17:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Doing some hacking on a flight to LA. (941 bytes, application/x-shellscript)
2016-01-18 08:10 EST, Daniel Walsh
no flags Details

  None (edit)
Description Jan Pazdziora 2015-11-20 03:03:10 EST
Description of problem:

When initscripts is installed in a Docker systemd container (typically via some dependency), running systemctl daemon-reload shows

Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
Failed to create unit file /run/systemd/generator.late/network.service: File exists

on the console.

Version-Release number of selected component (if applicable):

systemd-219-11.el7.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. Build image using the following Dockerfile:

FROM rhel7.2
RUN yum install -y initscripts && yum clean all
VOLUME [ "/tmp", "/run", "/data" ]
CMD [ "/usr/sbin/init" ]
LABEL RUN "docker run -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro ${NAME}"

2. Run it as docker run -ti --name systemd -v /sys/fs/cgroup:/sys/fs/cgroup:ro systemd
3. Withing the container, run systemctl daemon-reload: docker exec systemd systemctl daemon-reload

Actual results:

Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
Failed to create unit file /run/systemd/generator.late/network.service: File exists

in the terminal where the container was run.

Expected results:

No error.

Additional info:

In the journal, the following errors that strace suggests are related exist as well:

Nov 20 02:59:46 f65687dd1244 systemd[1]: Reloading.
Nov 20 02:59:46 f65687dd1244 systemd[1]: Attempted to remove disk file system, and we can't allow that.
Nov 20 02:59:46 f65687dd1244 systemd[1]: Attempted to remove disk file system, and we can't allow that.

When initscripts or /etc/rc.d/init.d/netconsole are not in the image, the issue is not present.
Comment 2 Lukáš Nykrýn 2015-11-23 08:35:07 EST
Yet another symptom of known incompatibles between docker and systemd. Systemd requires /run to be on tmpfs. 

Internal rm_rf function by default refuses to remove any file if it is not on tmpfs (because if you/we sc... something up, than you can fixit by reboot).

And on reload we delete the generators dirs and run generators again.
Comment 3 Lukáš Nykrýn 2016-01-12 09:26:19 EST
We require /run on tmpfs.
Comment 4 Jan Pazdziora 2016-01-13 02:20:43 EST
(In reply to Lukáš Nykrýn from comment #3)
> We require /run on tmpfs.

Or we need the assumption on systemd side relaxed, potentially based on the container environment variable.
Comment 5 Daniel Walsh 2016-01-13 08:35:01 EST
I believe we need to fix the docker environment to provide systemd the environment it expects.  We have most of this done, but are hung up on handling of the cgroup file system inside of the container.

In docker-1.10  you will be able to do something like


docker run -d --stop-signal=RTMIN+3  --tmpfs /run --tmpfs /tmp -v /sys/fs/cgroup:/sys/fs/cgroup:ro systemdcontainer

This will get you most of the way there, the container will be able to run without privileged I believe.  The problem with this is the cgroup config is writable so systemd will attempt to write to it, and be blocked by SELinux.  In a perfect world, we could mount the cgroups read/only or in the future their will be namespaced cgroups so the systemd inside of the container will only get a subsection of the cgroup controls.

We are attempting to build a docker hook that will setup the tmpfs and cgroups outside of user control.
Comment 6 Jan Pazdziora 2016-01-18 03:59:53 EST
(In reply to Daniel Walsh from comment #5)
> I believe we need to fix the docker environment to provide systemd the
> environment it expects.  We have most of this done, but are hung up on
> handling of the cgroup file system inside of the container.
> 
> In docker-1.10  you will be able to do something like
> 
> docker run -d --stop-signal=RTMIN+3  --tmpfs /run --tmpfs /tmp -v
> /sys/fs/cgroup:/sys/fs/cgroup:ro systemdcontainer

I wonder, how should things be configured for atomic (command) in compatible ways? When we start putting --tmpfs to LABELs, the docker run's will start failing when run on older dockers.
Comment 7 Daniel Walsh 2016-01-18 08:10 EST
Created attachment 1115849 [details]
Doing some hacking on a flight to LA.

Using this script will get you a little further, you could add the --tmpfs /run switch and I think systemd will come up a little more securely.  Sadly this will not get you fully where you want.

Adding oci-systemd and oci-register-machine hooks with our docker-1.10 get me to the state where systemd runs in a non --privileged container and

journalctl -M CONTINAERID

Works outside of the container.  Some how getting this to happen all under the covers would be my nirvana.
Comment 8 Daniel Walsh 2016-01-18 08:11:38 EST
I don't we should add these to tools until docker-1.10.  We might want to add something to the LABEL spec to state prereqs for containers, As we slowly build a new version of rpm.
Comment 9 Daniel Walsh 2016-02-22 14:00:34 EST
oci-register-machine got into rawhide today.  Next step is oci-systemd-hook.
Comment 10 Daniel Walsh 2016-06-03 15:44:57 EDT
Fixed in docker-1.10.
Comment 11 Jan Pazdziora 2016-06-07 12:48:58 EDT
With docker-1.10.3-31.el7.x86_64 and oci-register-machine-1.10.3-31.el7.x86_64 and oci-systemd-hook-1.10.3-31.el7.x86_64 installed, I still see the

Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists
Failed to create unit file /run/systemd/generator.late/network.service: File exists

errors when running commands from commment 0.
Comment 12 Daniel Walsh 2016-06-07 14:25:44 EDT
Any idea why?  What does the dockerfile look like.  Does this work in permissive mode?
Comment 13 Jan Pazdziora 2016-06-08 02:43:11 EDT
The problem is present with Dockerfile

FROM rhel7.2
RUN yum install -y --disablerepo='*-rpms' --enablerepo=rhel-7-server-rpms initscripts && yum clean all
VOLUME [ "/tmp", "/run" ]
CMD [ "/usr/sbin/init" ]
LABEL RUN "docker run -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro ${NAME}"

and is caused by the fact that the VOLUME definition in the Dockerfile prevents the oci-systemd-hook to mount /tmp and /run as tmpfs -- they are xfs.

When I comment out the

VOLUME [ "/tmp", "/run" ]

line, the problem disappears.

So this looks like it becomes an instance of bug 1331781.
Comment 14 Daniel Walsh 2016-06-08 12:12:20 EDT
Yup, but I would just suggest that we remove those.

Also with oci-systemd-hook you should not need  -v /sys/fs/cgroup:/sys/fs/cgroup:ro
Comment 16 Luwen Su 2016-06-11 12:51:38 EDT
In docker-1.10.3-40.el7.x86_64 , works well

# cat Dockerfile 
FROM fedora
RUN yum install -y  initscripts && yum clean all
CMD [ "/usr/sbin/init" ]
LABEL RUN "docker run -ti ${NAME}"

# docker build . 
# docker run -it --name systemd -v /sys/fs/cgroup:/sys/fs/cgroup:ro 5aefc25d4331

# docker exec systemd systemctl daemon-reload
Comment 18 errata-xmlrpc 2016-06-23 12:17:36 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1274

Note You need to log in before you can comment on or make changes to this bug.