Bug 1461571 - unable to run image built from 'docker.io/httpd'; 'could not open error log file /proc/self/fd/2'
Summary: unable to run image built from 'docker.io/httpd'; 'could not open error log f...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: container-selinux
Version: 25
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Lokesh Mandvekar
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1460341
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-06-14 20:00 UTC by Micah Abbott
Modified: 2017-07-25 00:23 UTC (History)
8 users (show)

Fixed In Version: container-selinux-2.20-2.fc26 container-selinux-2.20-2.fc25 container-selinux-2.21-1.fc26 container-selinux-2.21-1.fc25
Clone Of: 1460341
Environment:
Last Closed: 2017-07-24 19:20:47 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Micah Abbott 2017-06-14 20:00:22 UTC
Same problem exists in Fedora 25 Atomic Host:

# docker ps -a
CONTAINER ID        IMAGE               COMMAND              CREATED             STATUS                     PORTS               NAMES
57a50a35fdfc        apache_httpd        "httpd-foreground"   3 minutes ago       Exited (1) 2 minutes ago                       apache_httpd

# docker logs 57a50a35fdfc
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
(13)Permission denied: AH00091: httpd: could not open error log file /proc/self/fd/2.
AH00015: Unable to open logs
[root@micah-f25ah-vm0613a ~]# journalctl -b | grep -i avc
Jun 14 19:54:25 micah-f25ah-vm0613a.localdomain audit[6381]: AVC avc:  denied  { open } for  pid=6381 comm="httpd" path="pipe:[57170]" dev="pipefs" ino=57170 scontext=system_u:system_r:container_t:s0:c64,c482 tcontext=system_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=fifo_file permissive=0

# rpm -q docker container-selinux
docker-1.12.6-6.gitae7d637.fc25.x86_64
container-selinux-2.14-1.fc25.noarch

# atomic host status
State: idle
Deployments:
● fedora-atomic:fedora-atomic/25/x86_64/docker-host
             Version: 25.137 (2017-06-04 23:31:40)
              Commit: 0ed61d7441eddf96e6a98de4f10f4675268c7888b6d2b8a405b8c21fe6c92d23



+++ This bug was initially created as a clone of Bug #1460341 +++

One of our automated tests that builds 'httpd' images using various base images found a problem when trying to run an image built from 'docker.io/httpd'.  

This was testing against RHELAH 7.3.6 with 'docker-1.12.6-31.git3a6eaeb.el7.x86_64'


Using the following Dockerfile:

https://github.com/projectatomic/atomic-host-tests/blob/master/tests/docker-build-httpd/files/apache_httpd/Dockerfile

The image was built with:

# docker build --pull --rm -t apache_httpd -f Dockerfile .

Then run with:

# docker run -d -p 80:80 --name apache_httpd apache_httpd

But the container exits immediately.  The 'docker logs' show:

# docker logs apache_httpd
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
(13)Permission denied: AH00091: httpd: could not open error log file /proc/self/fd/2.
AH00015: Unable to open logs



This same container runs fine on RHELAH 7.3.5 using 'docker-1.12.6-28.git1398f24.el7.x86_64'


See the log below showing the results with both versions:


# atomic host status
State: idle
Deployments:
● 7.3_latest:rhel-atomic-host/7/x86_64/standard
             Version: 7.3.5 (2017-05-22 22:00:44)
              Commit: 0ccf9138962e5c2c3794969a228e751d13bb780f5b0a1f15f4a9649df06ba80a

# rpm -q docker
docker-1.12.6-28.git1398f24.el7.x86_64

# docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
apache_httpd                              latest              a251680b211f        17 minutes ago      177.5 MB
docker.io/httpd                           latest              e0645af13ada        4 weeks ago         177.5 MB

# docker run -d -p 80:80 --name apache_httpd apache_httpd
37c293f787b8360e774047f118f22c5f65defdd52003816abf9116d569e59a57

# docker ps -a
CONTAINER ID        IMAGE               COMMAND              CREATED             STATUS              PORTS                NAMES
37c293f787b8        apache_httpd        "httpd-foreground"   7 seconds ago       Up 5 seconds        0.0.0.0:80->80/tcp   apache_httpd

# curl http://localhost:80
SUCCESS apache_httpd


....upgrade/reboot....


# atomic host status
State: idle
Deployments:
● 7.3_latest:rhel-atomic-host/7/x86_64/standard
             Version: 7.3.6 (2017-06-07 21:38:15)
              Commit: a88c603af2a5c6b052b32c12b92c5578f71a8088e077781d17330275c63d03bd

  7.3_latest:rhel-atomic-host/7/x86_64/standard
             Version: 7.3.5 (2017-05-22 22:00:44)
              Commit: 0ccf9138962e5c2c3794969a228e751d13bb780f5b0a1f15f4a9649df06ba80a

# rpm -q docker
docker-1.12.6-31.git3a6eaeb.el7.x86_64

# docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
apache_httpd                              latest              a251680b211f        21 minutes ago      177.5 MB
docker.io/httpd                           latest              e0645af13ada        4 weeks ago         177.5 MB

# docker run -d -p 80:80 --name apache_httpd apache_httpd
1e12902555a72e0839ae68ac07ee9b1ee2e853d2704abe1e3fafaf059b0dd16a

# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

# docker ps -a
CONTAINER ID        IMAGE               COMMAND              CREATED             STATUS                     PORTS               NAMES
1e12902555a7        apache_httpd        "httpd-foreground"   8 seconds ago       Exited (1) 7 seconds ago                       apache_httpd

# docker logs apache_httpd 
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
(13)Permission denied: AH00091: httpd: could not open error log file /proc/self/fd/2.
AH00015: Unable to open logs

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-06-09 14:23:41 EDT ---

Since this bug report was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-06-09 14:23:41 EDT ---

This bug report has Keywords: Regression or TestBlocker.

Since no regressions or test blockers are allowed between releases, it is also being [proposed|marked] as a blocker for this release.

Please resolve ASAP.

--- Additional comment from Antonio Murdaca on 2017-06-09 14:28:05 EDT ---

Can you disable selinux and re-try?

--- Additional comment from Micah Abbott on 2017-06-09 15:22:19 EDT ---

Yeah, disabling SELinux works around it.

--- Additional comment from Micah Abbott on 2017-06-09 15:27:42 EDT ---

Similar problems with a 'httpd' container based on 'docker.io/nginx'

Build with this Dockerfile:

https://github.com/projectatomic/atomic-host-tests/blob/master/tests/docker-build-httpd/files/nginx_httpd/Dockerfile

The logs from that container:

# docker logs nginx_httpd 
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2017/06/09 19:14:34 [emerg] 1#1: open() "/var/log/nginx/error.log" failed (13: Permission denied)


Disabling SELinux in this case also worked around the error.

--- Additional comment from Daniel Walsh on 2017-06-10 08:20:51 EDT ---

I believe this is an issue in the latest container-selinux package.


Whenever you report an error with SELinux always attach the AVC messages.

ausearch -m avc -ts recent.



Also give me

ps -eZ | grep docker


I suspect that docker is running with something like
container_runtime_t:s0-s0:c10123

It used to run as
container_runtime_t:s0


I believe this is related to a problem we have seen in Fedora about docker leaking fds into the container to be used for stdin, stdout and stderr. I changed the way that the container runtimes run which is causing a new SELinux issue.

--- Additional comment from Micah Abbott on 2017-06-12 09:42:18 EDT ---

There's no `ausearch` on RHELAH, so the best I can give you is some grepped journal entries.


# journalctl -b | grep -e avc -e audit -e denied
Jun 12 13:39:56 micah-rhelah-vm0609a.localdomain kernel: type=1401 audit(1497274796.569:12): op=security_compute_av reason=bounds scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:svirt_lxc_net_t:s0-s0:c0.c1023 tclass=process perms=transition,sigchld,sigstop,signull,signal,getattr
Jun 12 13:39:56 micah-rhelah-vm0609a.localdomain kernel: type=1401 audit(1497274796.603:13): op=security_compute_av reason=bounds scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:svirt_lxc_net_t:s0-s0:c0.c1023 tclass=process perms=transition,sigchld,sigstop,signull,signal,getattr
Jun 12 13:40:29 micah-rhelah-vm0609a.localdomain kernel: type=1400 audit(1497274829.974:14): avc:  denied  { open } for  pid=14526 comm="httpd" path="pipe:[78235]" dev="pipefs" ino=78235 scontext=system_u:system_r:svirt_lxc_net_t:s0:c533,c834 tcontext=system_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=fifo_file
Jun 12 13:40:29 micah-rhelah-vm0609a.localdomain dockerd-current[5726]: (13)Permission denied: AH00091: httpd: could not open error log file /proc/self/fd/2.



# ps -eZ | grep docker
system_u:system_r:container_runtime_t:s0-s0:c0.c1023 5726 ? 00:00:45 dockerd-current
system_u:system_r:container_runtime_t:s0-s0:c0.c1023 5765 ? 00:00:03 docker-containe

--- Additional comment from Daniel Walsh on 2017-06-14 14:41:32 EDT ---

grep avc /var/log/audit/audit.log

--- Additional comment from Micah Abbott on 2017-06-14 15:37:49 EDT ---

(In reply to Daniel Walsh from comment #8)
> grep avc /var/log/audit/audit.log

That doesn't exist on RHELAH systems.

--- Additional comment from Daniel Walsh on 2017-06-14 15:39:17 EDT ---

Yes well you gave me the AVC's anyways.

journalctl -b | grep -i avc

:^)

--- Additional comment from Micah Abbott on 2017-06-14 15:39:31 EDT ---

(In reply to Micah Abbott from comment #9)
> (In reply to Daniel Walsh from comment #8)
> > grep avc /var/log/audit/audit.log
> 
> That doesn't exist on RHELAH systems.

Well, I won't make a blanket statement.  But it doesn't exist on the system I reproduced this on.

Comment 1 Daniel Walsh 2017-06-14 20:02:29 UTC
Fixed in container-selinux-2.19

Comment 2 Fedora Update System 2017-07-06 11:10:56 UTC
container-selinux-2.20-2.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-57cdd5b21d

Comment 3 Fedora Update System 2017-07-06 11:11:03 UTC
container-selinux-2.20-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-bfb1bf854d

Comment 4 Fedora Update System 2017-07-06 18:23:32 UTC
container-selinux-2.20-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-bfb1bf854d

Comment 5 Fedora Update System 2017-07-07 09:05:25 UTC
container-selinux-2.20-2.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-57cdd5b21d

Comment 6 Fedora Update System 2017-07-14 12:12:57 UTC
container-selinux-2.21-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6447e775e6

Comment 7 Fedora Update System 2017-07-14 12:13:09 UTC
container-selinux-2.21-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-1638ff6c79

Comment 8 Fedora Update System 2017-07-14 13:23:04 UTC
container-selinux-2.20-2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 9 Fedora Update System 2017-07-15 19:52:56 UTC
container-selinux-2.20-2.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 10 Fedora Update System 2017-07-15 21:51:29 UTC
container-selinux-2.21-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6447e775e6

Comment 11 Fedora Update System 2017-07-16 21:20:32 UTC
container-selinux-2.21-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-1638ff6c79

Comment 12 Fedora Update System 2017-07-24 19:20:47 UTC
container-selinux-2.21-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 13 Fedora Update System 2017-07-25 00:23:36 UTC
container-selinux-2.21-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.