Description of problem: When running systemd in a container, the pid 1's stdin, stdout, and stderr are now /dev/null. Version-Release number of selected component (if applicable): systemd-231-10.fc25.x86_64 How reproducible: Deterministic. Steps to Reproduce: 1. Have Dockerfile FROM fedora:25 COPY init /usr/local/sbin/ COPY test-fd /usr/sbin/ RUN chmod +x /usr/local/sbin/init /usr/sbin/test-fd COPY test-fd.service /usr/lib/systemd/system/ RUN ln -s /usr/lib/systemd/system/test-fd.service /usr/lib/systemd/system/default.target.wants/ ENV container docker ENTRYPOINT /usr/local/sbin/init 2. Have file init: #!/bin/bash ( trap '' SIGHUP SIGTERM tail --silent -n 0 -f --retry /var/log/out.log 2> /dev/null < /dev/null & ) exec /usr/sbin/init 3. Have file test-fd: #!/bin/bash ls -la /proc/1/fd > /var/log/out.log exec >> /proc/1/fd/1 2>> /proc/1/fd/2 echo "echo to stdout (/proc/1/fd/1)" 4. Have file test-fd.service: [Unit] Description=Test systemd fd [Service] Type=oneshot ExecStart=/usr/sbin/test-fd 5. Build container image: docker build -t test-fd . 6. Run container: docker run --tmpfs /run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name test-fd --rm -ti test-fd Actual results: lrwx------. 1 root root 64 Jan 13 15:30 0 -> /dev/null lrwx------. 1 root root 64 Jan 13 15:30 1 -> /dev/null [...] lrwx------. 1 root root 64 Jan 13 15:30 2 -> /dev/null and no echo to stdout (/proc/1/fd/1) line Expected results: With fedora:24 base image which has systemd-229-8.fc24.x86_64, there will be Starting Test systemd fd... [ OK ] Started Permit User Sessions. echo to stdout (/proc/1/fd/1) [ OK ] Started Test systemd fd. [ OK ] Reached target Multi-User System. [...] lrwx------. 1 root root 64 Jan 13 15:20 0 -> /2 lrwx------. 1 root root 64 Jan 13 15:20 1 -> /2 [...] lrwx------. 1 root root 64 Jan 13 15:20 2 -> /2 Additional info:
In 229, the logic in src/core/main.c was straightforward, not executed in containers: if (getpid() == 1 && detect_container() <= 0) { /* Running outside of a container as PID 1 */ arg_running_as = MANAGER_SYSTEM; make_null_stdio(); The git history shows multiple changes moving make_null_stdio around, as well as detect_container and attempts to do early bailout from fixup_environment in containers but it seems in 231 and HEAD, make_null_stdio is called even in containers. This breaks any systemd-based container which outputs from services to /proc/1/fd/1 to get output out of (docker) containers. Please fix the logic not to null stdio in containers and backport to Fedora 25.
Nothing has changed from the previous report. Systemd upstream thinks current behaviour is correct (https://github.com/systemd/systemd/pull/4262), and the docker bug is still open (https://github.com/docker/docker/issues/27202), docker guys are mulling it over. *** This bug has been marked as a duplicate of bug 1373780 ***
Reopening. The current behaviour of sytemd in docker is broken, and regression, and it breaks other applications, setups, and use-cases. I don't really care whose issue it is. If the patch with if (detect_container() <= 0) make_null_stdio(); is not acceptable for systemd upstream, let's carry it in Fedora only because we also carry docker of particular version which is not compatible. Or let's check for the container being docker, if you will, before docker gets fixed. But I'm really tired of seeing buggy systemd behaviour in containers and not seeing any drive to resolution, and users and admin taking the hit for putting workaround after workaround to their Dockerfiles and artifacts.
Also note that I don't try to use /dev/console at all in this bugzilla, so I don't believe it should be marked as dupe of bug 1373780 just yet.
Also note that L.P.'s comment https://github.com/systemd/systemd/pull/4262#issuecomment-254887097 says again, docker shouldn't close the pty for good when it sees POLLHUP on it. That's all there's to fix... It simply does not apply for this bugzilla because with the - make_null_stdio(); + if (detect_container() <= 0) + make_null_stdio(); I do /proc/1/fd/1 working even if I invoke docker run with -ti ... IOW, when there is not pty there, just plain stdin/out/err. Please add the patch to Fedora 25+ builds of systemd.
(In reply to Jan Pazdziora from comment #5) > I do /proc/1/fd/1 working even if I invoke docker run with -ti ... IOW, Sorry for the confusion, this should read: I see /proc/1/fd/1 working even if I invoke docker run without -ti.
I've reopened upstream PR that introduces docker specific workaround in order to fix this issue. https://github.com/systemd/systemd/pull/4262 @zbyszek, would you be ok with providing this workaround downstream only in case that upstream again decides to not merge it for whatever reason?
The issues should first come to conclusion upstream. Currently the last comment is https://github.com/systemd/systemd/pull/4262#issuecomment-278093483: > please provide a proper, new PR that actually does the right thing, and also has the effect that we actually use STDERR for logging in this case. Otherwise, we'll keep the fd open, but then log with another fd. Which is broken. > Also, the patch this PR is about doesn't even do what it is supposed to do and isn't mergable due to conflicts.
In the meantime I talked to some RH people who work on docker and they promised to get the bug fixed on docker side. I will talk to them again in two weeks. In case they don't have the fix ready by then I will go ahead and resubmit systemd workaround.
Any progress with the fix?
(In reply to Jan Pazdziora from comment #10) > Any progress with the fix? There was some progress recently on this. This is the newest attempt to fix this on runc/docker side, https://github.com/opencontainers/runc/pull/1446 Apparently, kernel delivers POLLHUP (translated to EOF as seen by Go's io.Copy() routine) on master fd only when last fd referencing slave is closed. They now workaround this by duping slave fd in the container and send that duplicate slave fd back to parent runc process. Hence parent holds reference to slave pty end during entire lifetime of the container. Thus it doesn't matter if PID 1 in the container closes its end. As for fix on systemd's side. Lennart changed his mind about this and doesn't want to put in any workarounds to systemd. He argues that systemd does the right thing and container runtime should be able to handle it.
(In reply to Michal Sekletar from comment #11) > > As for fix on systemd's side. Lennart changed his mind about this and > doesn't want to put in any workarounds to systemd. He argues that systemd > does the right thing and container runtime should be able to handle it. The fact it though that systemd changed its behaviour against what we had in fedora:24 image, so the interoperability regression was introduced by change in systemd. Can we carry the patch in Fedora's systemd package?
Please also note that while https://github.com/opencontainers/runc/pull/1446 talks about "Currently when terminal=true", the problem in this bugzilla is present even if you do not use the -t option to docker run. It does not seem to be related to the terminal / console issue discussed elsewhere.
Any news about this issue and how the systemd team will do end-to-end testing in containerized environments to maintain functionality which worked in the past?
This issue really blocks any automated testing of systemd in Fedora 25 or newer containers. Do you see any chance of getting the issue investigated and resolved?
This message is a reminder that Fedora 25 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 25. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '25'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 25 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
The issue is still present with fedora:26, fedora:27, and fedora:rawhide.
This message is a reminder that Fedora 26 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '26'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 26 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
The issues is still present with REPOSITORY TAG IMAGE ID CREATED SIZE registry.fedoraproject.org/fedora 27 6e2c293f3942 2 weeks ago 235 MB which contains systemd-234-10.git5f8984e.fc27.x86_64.
(In reply to Jan Pazdziora from comment #14) > Any news about this issue and how the systemd team will do end-to-end > testing in containerized environments to maintain functionality which worked > in the past? systemd upstream does testing of containerized environments using systemd-nspawn. Testing using docker is out-of-scope for now. Also at this point it probably makes more sense to start focusing on testing using podman.
Will the systemd upstream or Fedora maintainers start testing containerized environments using podman? 'cause I don't want to get to the same situations again when it feels like I'm filing bugs and nobody cares.
(In reply to Jan Pazdziora from comment #21) > Will the systemd upstream or Fedora maintainers start testing containerized > environments using podman? 'cause I don't want to get to the same situations > again when it feels like I'm filing bugs and nobody cares. We should be able to do this now on Fedora side quite easily in Fedora CI infra. Do you have any specific use case in mind? I think we should probably start with just booting the default Fedora container image.
Right. And checking status of services and overall status of systemd, and grepping journalctl output for any mentions of errors and failures. When you get that enabled, please show the link to the definition here and we can then start adding some more specific tests for issues that I've seen failing.
This message is a reminder that Fedora 27 is nearing its end of life. On 2018-Nov-30 Fedora will stop maintaining and issuing updates for Fedora 27. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '27'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 27 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
The issue is still present with systemd-239-3.fc29.x86_64 on container image registry.fedoraproject.org/fedora:29. Tested on Fedora 28 host with docker-1.13.1-61.git9cb56fd.fc28.x86_64.
This message is a reminder that Fedora 29 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '29'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 29 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
The issues is still present with systemd-241-12.git323cdf4.fc30.x86_64 in container images registry.fedoraproject.org/fedora:30. Tested on Fedora 31 host with docker-1.13.1-68.git47e2230.fc30.x86_64.
This message is a reminder that Fedora 30 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '30'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 30 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 30 changed to end-of-life (EOL) status on 2020-05-26. Fedora 30 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.