Bug 1413099 - systemd makes its stdio /dev/null when running under container=docker
Summary: systemd makes its stdio /dev/null when running under container=docker
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 30
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-13 15:47 UTC by Jan Pazdziora
Modified: 2020-05-26 17:44 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-26 17:44:31 UTC
Type: Bug


Attachments (Terms of Use)

Description Jan Pazdziora 2017-01-13 15:47:15 UTC
Description of problem:

When running systemd in a container, the pid 1's stdin, stdout, and stderr are now /dev/null.

Version-Release number of selected component (if applicable):

systemd-231-10.fc25.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. Have Dockerfile

FROM fedora:25
COPY init /usr/local/sbin/
COPY test-fd /usr/sbin/
RUN chmod +x /usr/local/sbin/init /usr/sbin/test-fd
COPY test-fd.service /usr/lib/systemd/system/
RUN ln -s /usr/lib/systemd/system/test-fd.service /usr/lib/systemd/system/default.target.wants/
ENV container docker
ENTRYPOINT /usr/local/sbin/init

2. Have file init:

#!/bin/bash

(
trap '' SIGHUP SIGTERM
tail --silent -n 0 -f --retry /var/log/out.log 2> /dev/null < /dev/null &
)
exec /usr/sbin/init

3. Have file test-fd:

#!/bin/bash

ls -la /proc/1/fd > /var/log/out.log

exec >> /proc/1/fd/1 2>> /proc/1/fd/2

echo "echo to stdout (/proc/1/fd/1)"

4. Have file test-fd.service:

[Unit]
Description=Test systemd fd

[Service]
Type=oneshot
ExecStart=/usr/sbin/test-fd

5. Build container image: docker build -t test-fd .
6. Run container: docker run --tmpfs /run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name test-fd --rm -ti test-fd

Actual results:

lrwx------. 1 root root 64 Jan 13 15:30 0 -> /dev/null
lrwx------. 1 root root 64 Jan 13 15:30 1 -> /dev/null
[...]
lrwx------. 1 root root 64 Jan 13 15:30 2 -> /dev/null

and no

echo to stdout (/proc/1/fd/1)

line

Expected results:

With fedora:24 base image which has systemd-229-8.fc24.x86_64, there will be

         Starting Test systemd fd...
[  OK  ] Started Permit User Sessions.
echo to stdout (/proc/1/fd/1)
[  OK  ] Started Test systemd fd.
[  OK  ] Reached target Multi-User System.
[...]
lrwx------. 1 root root 64 Jan 13 15:20 0 -> /2
lrwx------. 1 root root 64 Jan 13 15:20 1 -> /2
[...]
lrwx------. 1 root root 64 Jan 13 15:20 2 -> /2

Additional info:

Comment 1 Jan Pazdziora 2017-01-13 15:53:46 UTC
In 229, the logic in src/core/main.c was straightforward, not executed in containers:

        if (getpid() == 1 && detect_container() <= 0) {

                /* Running outside of a container as PID 1 */
                arg_running_as = MANAGER_SYSTEM;
                make_null_stdio();

The git history shows multiple changes moving make_null_stdio around, as well as detect_container and attempts to do early bailout from fixup_environment in containers but it seems in 231 and HEAD, make_null_stdio is called even in containers.

This breaks any systemd-based container which outputs from services to /proc/1/fd/1 to get output out of (docker) containers.

Please fix the logic not to null stdio in containers and backport to Fedora 25.

Comment 2 Zbigniew Jędrzejewski-Szmek 2017-01-13 17:10:27 UTC
Nothing has changed from the previous report. Systemd upstream thinks current behaviour is correct (https://github.com/systemd/systemd/pull/4262), and the docker bug is still open (https://github.com/docker/docker/issues/27202), docker guys are mulling it over.

*** This bug has been marked as a duplicate of bug 1373780 ***

Comment 3 Jan Pazdziora 2017-01-16 10:54:09 UTC
Reopening.

The current behaviour of sytemd in docker is broken, and regression, and it breaks other applications, setups, and use-cases.

I don't really care whose issue it is. If the patch with

    if (detect_container() <= 0)
        make_null_stdio();

is not acceptable for systemd upstream, let's carry it in Fedora only because we also carry docker of particular version which is not compatible. Or let's check for the container being docker, if you will, before docker gets fixed.

But I'm really tired of seeing buggy systemd behaviour in containers and not seeing any drive to resolution, and users and admin taking the hit for putting workaround after workaround to their Dockerfiles and artifacts.

Comment 4 Jan Pazdziora 2017-01-16 10:55:20 UTC
Also note that I don't try to use /dev/console at all in this bugzilla, so I don't believe it should be marked as dupe of bug 1373780 just yet.

Comment 5 Jan Pazdziora 2017-01-16 11:03:01 UTC
Also note that L.P.'s comment https://github.com/systemd/systemd/pull/4262#issuecomment-254887097 says

    again, docker shouldn't close the pty for good when it sees POLLHUP on it.
    That's all there's to fix...

It simply does not apply for this bugzilla because with the

-                make_null_stdio();
+                if (detect_container() <= 0)
+                        make_null_stdio();

I do /proc/1/fd/1 working even if I invoke docker run with -ti ... IOW, when there is not pty there, just plain stdin/out/err.

Please add the patch to Fedora 25+ builds of systemd.

Comment 6 Jan Pazdziora 2017-01-16 11:07:06 UTC
(In reply to Jan Pazdziora from comment #5)
> I do /proc/1/fd/1 working even if I invoke docker run with -ti ... IOW,

Sorry for the confusion, this should read:

I see /proc/1/fd/1 working even if I invoke docker run without -ti.

Comment 7 Michal Sekletar 2017-02-07 16:50:44 UTC
I've reopened upstream PR that introduces docker specific workaround in order to fix this issue.

https://github.com/systemd/systemd/pull/4262

@zbyszek, would you be ok with providing this workaround downstream only in case that upstream again decides to not merge it for whatever reason?

Comment 8 Zbigniew Jędrzejewski-Szmek 2017-02-26 04:13:35 UTC
The issues should first come to conclusion upstream. Currently the last comment is https://github.com/systemd/systemd/pull/4262#issuecomment-278093483:

> please provide a proper, new PR that actually does the right thing, and also has the effect that we actually use STDERR for logging in this case. Otherwise, we'll keep the fd open, but then log with another fd. Which is broken.

> Also, the patch this PR is about doesn't even do what it is supposed to do and isn't mergable due to conflicts.

Comment 9 Michal Sekletar 2017-02-27 09:56:44 UTC
In the meantime I talked to some RH people who work on docker and they promised to get the bug fixed on docker side. I will talk to them again in two weeks. In case they don't have the fix ready by then I will go ahead and resubmit systemd workaround.

Comment 10 Jan Pazdziora 2017-05-15 10:05:22 UTC
Any progress with the fix?

Comment 11 Michal Sekletar 2017-05-15 12:11:48 UTC
(In reply to Jan Pazdziora from comment #10)
> Any progress with the fix?

There was some progress recently on this. This is the newest attempt to fix this on runc/docker side,

https://github.com/opencontainers/runc/pull/1446

Apparently, kernel delivers POLLHUP (translated to EOF as seen by Go's io.Copy() routine) on master fd only when last fd referencing slave is closed. They now workaround this by duping slave fd in the container and send that duplicate slave fd back to parent runc process. Hence parent holds reference to slave pty end during entire lifetime of the container. Thus it doesn't matter if PID 1 in the container closes its end.

As for fix on systemd's side. Lennart changed his mind about this and doesn't want to put in any workarounds to systemd. He argues that systemd does the right thing and container runtime should be able to handle it.

Comment 12 Jan Pazdziora 2017-05-15 12:25:15 UTC
(In reply to Michal Sekletar from comment #11)
> 
> As for fix on systemd's side. Lennart changed his mind about this and
> doesn't want to put in any workarounds to systemd. He argues that systemd
> does the right thing and container runtime should be able to handle it.

The fact it though that systemd changed its behaviour against what we had in fedora:24 image, so the interoperability regression was introduced by change in systemd.

Can we carry the patch in Fedora's systemd package?

Comment 13 Jan Pazdziora 2017-05-15 12:30:17 UTC
Please also note that while https://github.com/opencontainers/runc/pull/1446 talks about "Currently when terminal=true", the problem in this bugzilla is present even if you do not use the -t option to docker run. It does not seem to be related to the terminal / console issue discussed elsewhere.

Comment 14 Jan Pazdziora 2017-09-22 07:15:33 UTC
Any news about this issue and how the systemd team will do end-to-end testing in containerized environments to maintain functionality which worked in the past?

Comment 15 Jan Pazdziora 2017-10-17 10:18:59 UTC
This issue really blocks any automated testing of systemd in Fedora 25 or newer containers. Do you see any chance of getting the issue investigated and resolved?

Comment 16 Fedora End Of Life 2017-11-16 19:04:23 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 17 Jan Pazdziora 2017-11-29 14:44:26 UTC
The issue is still present with fedora:26, fedora:27, and fedora:rawhide.

Comment 18 Fedora End Of Life 2018-05-03 07:59:07 UTC
This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 19 Jan Pazdziora 2018-05-09 09:41:43 UTC
The issues is still present with

REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
registry.fedoraproject.org/fedora   27                  6e2c293f3942        2 weeks ago         235 MB

which contains systemd-234-10.git5f8984e.fc27.x86_64.

Comment 20 Michal Sekletar 2018-10-23 07:34:11 UTC
(In reply to Jan Pazdziora from comment #14)
> Any news about this issue and how the systemd team will do end-to-end
> testing in containerized environments to maintain functionality which worked
> in the past?

systemd upstream does testing of containerized environments using systemd-nspawn. Testing using docker is out-of-scope for now. Also at this point it probably makes more sense to start focusing on testing using podman.

Comment 21 Jan Pazdziora 2018-10-23 07:42:45 UTC
Will the systemd upstream or Fedora maintainers start testing containerized environments using podman? 'cause I don't want to get to the same situations again when it feels like I'm filing bugs and nobody cares.

Comment 22 Michal Sekletar 2018-10-23 11:30:03 UTC
(In reply to Jan Pazdziora from comment #21)
> Will the systemd upstream or Fedora maintainers start testing containerized
> environments using podman? 'cause I don't want to get to the same situations
> again when it feels like I'm filing bugs and nobody cares.

We should be able to do this now on Fedora side quite easily in Fedora CI infra. 

Do you have any specific use case in mind? I think we should probably start with just booting the default Fedora container image.

Comment 23 Jan Pazdziora 2018-10-23 11:44:25 UTC
Right. And checking status of services and overall status of systemd, and grepping journalctl output for any mentions of errors and failures. When you get that enabled, please show the link to the definition here and we can then start adding some more specific tests for issues that I've seen failing.

Comment 24 Ben Cotton 2018-11-27 13:57:20 UTC
This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 25 Ben Cotton 2018-11-30 23:18:34 UTC
Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 26 Jan Pazdziora 2018-12-13 11:46:50 UTC
The issue is still present with systemd-239-3.fc29.x86_64 on container image registry.fedoraproject.org/fedora:29.

Tested on Fedora 28 host with docker-1.13.1-61.git9cb56fd.fc28.x86_64.

Comment 27 Ben Cotton 2019-10-31 19:14:51 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 28 Jan Pazdziora 2019-11-01 10:09:58 UTC
The issues is still present with systemd-241-12.git323cdf4.fc30.x86_64 in container images registry.fedoraproject.org/fedora:30.

Tested on Fedora 31 host with docker-1.13.1-68.git47e2230.fc30.x86_64.

Comment 29 Ben Cotton 2020-04-30 20:47:56 UTC
This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 30 Ben Cotton 2020-05-26 17:44:31 UTC
Fedora 30 changed to end-of-life (EOL) status on 2020-05-26. Fedora 30 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.