Bug 1283891
| Summary: | When initscripts is installed, systemd in Docker container shows error during systemctl daemon-reload | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Jan Pazdziora (Red Hat) <jpazdziora> | ||||
| Component: | docker | Assignee: | Daniel Walsh <dwalsh> | ||||
| Status: | CLOSED ERRATA | QA Contact: | atomic-bugs <atomic-bugs> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.2 | CC: | jpazdziora, lnykryn, lsm5, lsu, sdodson, systemd-maint-list | ||||
| Target Milestone: | rc | Keywords: | Extras | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-06-23 16:17:36 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1283777 | ||||||
| Attachments: |
|
||||||
Yet another symptom of known incompatibles between docker and systemd. Systemd requires /run to be on tmpfs. Internal rm_rf function by default refuses to remove any file if it is not on tmpfs (because if you/we sc... something up, than you can fixit by reboot). And on reload we delete the generators dirs and run generators again. We require /run on tmpfs. (In reply to Lukáš Nykrýn from comment #3) > We require /run on tmpfs. Or we need the assumption on systemd side relaxed, potentially based on the container environment variable. I believe we need to fix the docker environment to provide systemd the environment it expects. We have most of this done, but are hung up on handling of the cgroup file system inside of the container. In docker-1.10 you will be able to do something like docker run -d --stop-signal=RTMIN+3 --tmpfs /run --tmpfs /tmp -v /sys/fs/cgroup:/sys/fs/cgroup:ro systemdcontainer This will get you most of the way there, the container will be able to run without privileged I believe. The problem with this is the cgroup config is writable so systemd will attempt to write to it, and be blocked by SELinux. In a perfect world, we could mount the cgroups read/only or in the future their will be namespaced cgroups so the systemd inside of the container will only get a subsection of the cgroup controls. We are attempting to build a docker hook that will setup the tmpfs and cgroups outside of user control. (In reply to Daniel Walsh from comment #5) > I believe we need to fix the docker environment to provide systemd the > environment it expects. We have most of this done, but are hung up on > handling of the cgroup file system inside of the container. > > In docker-1.10 you will be able to do something like > > docker run -d --stop-signal=RTMIN+3 --tmpfs /run --tmpfs /tmp -v > /sys/fs/cgroup:/sys/fs/cgroup:ro systemdcontainer I wonder, how should things be configured for atomic (command) in compatible ways? When we start putting --tmpfs to LABELs, the docker run's will start failing when run on older dockers. Created attachment 1115849 [details]
Doing some hacking on a flight to LA.
Using this script will get you a little further, you could add the --tmpfs /run switch and I think systemd will come up a little more securely. Sadly this will not get you fully where you want.
Adding oci-systemd and oci-register-machine hooks with our docker-1.10 get me to the state where systemd runs in a non --privileged container and
journalctl -M CONTINAERID
Works outside of the container. Some how getting this to happen all under the covers would be my nirvana.
I don't we should add these to tools until docker-1.10. We might want to add something to the LABEL spec to state prereqs for containers, As we slowly build a new version of rpm. oci-register-machine got into rawhide today. Next step is oci-systemd-hook. Fixed in docker-1.10. With docker-1.10.3-31.el7.x86_64 and oci-register-machine-1.10.3-31.el7.x86_64 and oci-systemd-hook-1.10.3-31.el7.x86_64 installed, I still see the Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists Failed to create unit file /run/systemd/generator.late/network.service: File exists errors when running commands from commment 0. Any idea why? What does the dockerfile look like. Does this work in permissive mode? The problem is present with Dockerfile
FROM rhel7.2
RUN yum install -y --disablerepo='*-rpms' --enablerepo=rhel-7-server-rpms initscripts && yum clean all
VOLUME [ "/tmp", "/run" ]
CMD [ "/usr/sbin/init" ]
LABEL RUN "docker run -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro ${NAME}"
and is caused by the fact that the VOLUME definition in the Dockerfile prevents the oci-systemd-hook to mount /tmp and /run as tmpfs -- they are xfs.
When I comment out the
VOLUME [ "/tmp", "/run" ]
line, the problem disappears.
So this looks like it becomes an instance of bug 1331781.
Yup, but I would just suggest that we remove those. Also with oci-systemd-hook you should not need -v /sys/fs/cgroup:/sys/fs/cgroup:ro In docker-1.10.3-40.el7.x86_64 , works well
# cat Dockerfile
FROM fedora
RUN yum install -y initscripts && yum clean all
CMD [ "/usr/sbin/init" ]
LABEL RUN "docker run -ti ${NAME}"
# docker build .
# docker run -it --name systemd -v /sys/fs/cgroup:/sys/fs/cgroup:ro 5aefc25d4331
# docker exec systemd systemctl daemon-reload
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1274 |
Description of problem: When initscripts is installed in a Docker systemd container (typically via some dependency), running systemctl daemon-reload shows Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists Failed to create unit file /run/systemd/generator.late/network.service: File exists on the console. Version-Release number of selected component (if applicable): systemd-219-11.el7.x86_64 How reproducible: Deterministic. Steps to Reproduce: 1. Build image using the following Dockerfile: FROM rhel7.2 RUN yum install -y initscripts && yum clean all VOLUME [ "/tmp", "/run", "/data" ] CMD [ "/usr/sbin/init" ] LABEL RUN "docker run -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro ${NAME}" 2. Run it as docker run -ti --name systemd -v /sys/fs/cgroup:/sys/fs/cgroup:ro systemd 3. Withing the container, run systemctl daemon-reload: docker exec systemd systemctl daemon-reload Actual results: Failed to create unit file /run/systemd/generator.late/netconsole.service: File exists Failed to create unit file /run/systemd/generator.late/network.service: File exists in the terminal where the container was run. Expected results: No error. Additional info: In the journal, the following errors that strace suggests are related exist as well: Nov 20 02:59:46 f65687dd1244 systemd[1]: Reloading. Nov 20 02:59:46 f65687dd1244 systemd[1]: Attempted to remove disk file system, and we can't allow that. Nov 20 02:59:46 f65687dd1244 systemd[1]: Attempted to remove disk file system, and we can't allow that. When initscripts or /etc/rc.d/init.d/netconsole are not in the image, the issue is not present.