Bug 1443922 - systemd fails to run in container due to mounting issues
Summary: systemd fails to run in container due to mounting issues
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: docker
Version: 26
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Antonio Murdaca
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-04-20 09:19 UTC by James Hogarth
Modified: 2017-06-24 03:05 UTC (History)
19 users (show)

Fixed In Version: docker-1.13.1-19.git27e468e.fc26
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-24 03:05:53 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github projectatomic oci-systemd-hook issues 56 0 'None' closed systemd fails to run in container due to mounting issues 2020-12-11 02:20:13 UTC

Description James Hogarth 2017-04-20 09:19:21 UTC
Description of problem:
When running a container systemd doesn't complete initialisation and systemctl cannot be used to start/stop services in the container, complaining of a dbus error.

Version-Release number of selected component (if applicable):
docker-1.13.1-7.git14cc629.fc26.x86_64
docker-client-4.0.6-5.fc26.noarch
docker-rhel-push-plugin-1.13.1-7.git14cc629.fc26.x86_64
docker-v1.10-migrator-1.13.1-7.git14cc629.fc26.x86_64
docker-common-1.13.1-7.git14cc629.fc26.x86_64
oci-systemd-hook-0.1.7-1.git1788cf2.fc26.x86_64

How reproducible:
Always

Steps to Reproduce:
1. mkdir httpd-test ; cd http-test
2. cat > Dockerfile <<EOF
FROM fedora:latest
ENV container oci
RUN dnf -y install httpd; dnf clean all ; systemctl enable httpd
STOPSIGNAL SIGRTMIN+3
EXPOSE 80
CMD ["/sbin/init"]
EOF
3. docker build -t http-test .
4. docker run -dt  --name http-test  http-test

Actual results:
docker logs http-test 
Failed to determine whether /sys is a mount point: Operation not permitted
Failed to determine whether /proc is a mount point: Operation not permitted
Failed to determine whether /dev is a mount point: Operation not permitted
Failed to determine whether /dev/shm is a mount point: Operation not permitted
Failed to determine whether /run is a mount point: Operation not permitted
Failed to determine whether /sys/fs/cgroup is a mount point: Operation not permitted
Failed to determine whether /sys/fs/cgroup/systemd is a mount point: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.


Expected results:
 docker top http-test 
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                11860               11843               0                   10:15               ?                   00:00:00            /sbin/init
root                11954               11860               0                   10:15               ?                   00:00:00            /usr/lib/systemd/systemd-journald
root                12154               11860               1                   10:15               ?                   00:00:00            /usr/sbin/httpd -DFOREGROUND
apache              12155               12154               0                   10:15               ?                   00:00:00            /usr/sbin/httpd -DFOREGROUND
apache              12156               12154               0                   10:15               ?                   00:00:00            /usr/sbin/httpd -DFOREGROUND
apache              12157               12154               0                   10:15               ?                   00:00:00            /usr/sbin/httpd -DFOREGROUND
apache              12162               12154               0                   10:15               ?                   00:00:00            /usr/sbin/httpd -DFOREGROUND
apache              12181               12154               0                   10:15               ?                   00:00:00            /usr/sbin/httpd -DFOREGROUND
dbus                12186               11860               0                   10:15               ?                   00:00:00            /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root                12188               11860               0                   10:16               ?                   00:00:00            /usr/lib/systemd/systemd-logind


Additional info:
The hook works with SYS_ADMIN caps enabled for the container.

docker run -dt  --cap-add SYS_ADMIN --name http-test  http-test

Comment 1 Daniel Walsh 2017-04-20 12:33:42 UTC
This is a systemd issue that is fixed in the upstream version of systemd, I believe.  The problem is seccomp removed a syscall that is causing systemd to be confused about itself running in a container, (I believe).

Comment 2 James Hogarth 2017-04-20 12:36:53 UTC
My test case fails whether the container is from centos:latest or fedora:latest

Unless you mean the host OS systemd being involved somehow?

Comment 3 Daniel Walsh 2017-04-20 12:39:04 UTC
Try running with --security-opt seccomp:unconfined

And see if it works.

Comment 4 James Hogarth 2017-04-20 13:00:43 UTC
You're right ... that works.

Comment 5 Daniel Walsh 2017-04-20 13:08:42 UTC
I think this can be fixed in two different ways, we can add back in the syscall that seccomp is blocking that is causing systemd to fail.  But I believe Lennart put in a fix for systemd that made systemd work without the syscall.

Comment 6 Daniel Walsh 2017-04-20 13:09:14 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=1441100

Talks about fixing the seccomp issue.

Comment 7 James Hogarth 2017-04-20 13:26:14 UTC
DO you know if on the RHEL side it was fixed with a patch to docker-latest to add the seccomp call back to the whitelist, or if the systemd patch was located and backported?

It'd make sense to use the same fix here in Fedora too

Comment 8 Daniel Walsh 2017-04-20 13:44:56 UTC
Agreed, basically this fix is just to add the syscall back into the syscall json file we ship.  The same code should be in Fedora as well as RHEL/Centos.

Comment 9 James Hogarth 2017-04-24 08:51:30 UTC
So does this mean this bug should actually be with docker not systemd for that particular fix?

Comment 10 Daniel Walsh 2017-04-24 12:43:32 UTC
I believe this is not a bug per se with container runtimes.  Container runtimes are attempting to run with as tight of security as possible.  systemd has a fix to be able to work fine under these tighter constraints so this systemd should get released into Fedora.  Since it takes a lot longer to get systemd updated in RHEL/Centos we will need to loosen the security constraints in container runtimes for now.

Comment 11 James Hogarth 2017-05-10 11:06:47 UTC
Dan this has been hanging around for a while now, can we get the temp fix in docker asap as we're drawing close to F26 beta and systemd containers on F26 are totally broken at present by default.

Comment 12 Daniel Walsh 2017-05-10 12:09:50 UTC
Lokesh or runcom can we build the docker package with the separate seccomp.json file which fixes this issue.

Comment 13 James Hogarth 2017-05-25 20:42:21 UTC
Lokesh it's been a fortnight since Dan reassigned and this is really painful to deal with... any chance of that seccomp json update rolling out soon?

Comment 14 Antonio Murdaca 2017-06-13 14:52:54 UTC
I'm rebuilding it.

Comment 15 Fedora Update System 2017-06-13 14:56:28 UTC
docker-1.13.1-17.git51eb16e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-84372bb930

Comment 16 Zbigniew Jędrzejewski-Szmek 2017-06-13 18:12:52 UTC
Daniel, you mentioned some communication with Lennart Poettering to make some changes in systemd. There seems to be no trace of this in either this bug or #1441100, and I'm not aware of any patches in systemd related to this.

The change to allow the syscall in docker seems like the proper fix, since this is a generally useful syscall, but if there's an different fix in systemd, I'd backport that too. So do you have any details?

Comment 17 Fedora Update System 2017-06-15 13:56:45 UTC
docker-1.13.1-17.git51eb16e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-84372bb930

Comment 18 Fedora Update System 2017-06-15 14:20:34 UTC
docker-1.13.1-18.git27e468e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-740382b040

Comment 19 Fedora Update System 2017-06-16 22:24:19 UTC
docker-1.13.1-18.git27e468e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-740382b040

Comment 20 Fedora Update System 2017-06-19 19:53:38 UTC
docker-1.13.1-19.git27e468e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-36d05500ee

Comment 21 Fedora Update System 2017-06-21 06:21:51 UTC
docker-1.13.1-19.git27e468e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-36d05500ee

Comment 22 Fedora Update System 2017-06-24 03:05:53 UTC
docker-1.13.1-19.git27e468e.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.