Bug 1460341

Summary: unable to run image built from 'docker.io/httpd'; 'could not open error log file /proc/self/fd/2'
Product: Red Hat Enterprise Linux 7 Reporter: Micah Abbott <miabbott>
Component: container-selinuxAssignee: Daniel Walsh <dwalsh>
Status: CLOSED ERRATA QA Contact: atomic-bugs <atomic-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: amurdaca, dwalsh, lsm5, miabbott
Target Milestone: rcKeywords: Extras, Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: container-selinux-2.19-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1461571 (view as bug list) Environment:
Last Closed: 2017-06-28 15:41:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1461571    

Description Micah Abbott 2017-06-09 18:23:33 UTC 
One of our automated tests that builds 'httpd' images using various base images found a problem when trying to run an image built from 'docker.io/httpd'.  

This was testing against RHELAH 7.3.6 with 'docker-1.12.6-31.git3a6eaeb.el7.x86_64'


Using the following Dockerfile:

https://github.com/projectatomic/atomic-host-tests/blob/master/tests/docker-build-httpd/files/apache_httpd/Dockerfile

The image was built with:

# docker build --pull --rm -t apache_httpd -f Dockerfile .

Then run with:

# docker run -d -p 80:80 --name apache_httpd apache_httpd

But the container exits immediately.  The 'docker logs' show:

# docker logs apache_httpd
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
(13)Permission denied: AH00091: httpd: could not open error log file /proc/self/fd/2.
AH00015: Unable to open logs



This same container runs fine on RHELAH 7.3.5 using 'docker-1.12.6-28.git1398f24.el7.x86_64'


See the log below showing the results with both versions:


# atomic host status
State: idle
Deployments:
● 7.3_latest:rhel-atomic-host/7/x86_64/standard
             Version: 7.3.5 (2017-05-22 22:00:44)
              Commit: 0ccf9138962e5c2c3794969a228e751d13bb780f5b0a1f15f4a9649df06ba80a

# rpm -q docker
docker-1.12.6-28.git1398f24.el7.x86_64

# docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
apache_httpd                              latest              a251680b211f        17 minutes ago      177.5 MB
docker.io/httpd                           latest              e0645af13ada        4 weeks ago         177.5 MB

# docker run -d -p 80:80 --name apache_httpd apache_httpd
37c293f787b8360e774047f118f22c5f65defdd52003816abf9116d569e59a57

# docker ps -a
CONTAINER ID        IMAGE               COMMAND              CREATED             STATUS              PORTS                NAMES
37c293f787b8        apache_httpd        "httpd-foreground"   7 seconds ago       Up 5 seconds        0.0.0.0:80->80/tcp   apache_httpd

# curl http://localhost:80
SUCCESS apache_httpd


....upgrade/reboot....


# atomic host status
State: idle
Deployments:
● 7.3_latest:rhel-atomic-host/7/x86_64/standard
             Version: 7.3.6 (2017-06-07 21:38:15)
              Commit: a88c603af2a5c6b052b32c12b92c5578f71a8088e077781d17330275c63d03bd

  7.3_latest:rhel-atomic-host/7/x86_64/standard
             Version: 7.3.5 (2017-05-22 22:00:44)
              Commit: 0ccf9138962e5c2c3794969a228e751d13bb780f5b0a1f15f4a9649df06ba80a

# rpm -q docker
docker-1.12.6-31.git3a6eaeb.el7.x86_64

# docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
apache_httpd                              latest              a251680b211f        21 minutes ago      177.5 MB
docker.io/httpd                           latest              e0645af13ada        4 weeks ago         177.5 MB

# docker run -d -p 80:80 --name apache_httpd apache_httpd
1e12902555a72e0839ae68ac07ee9b1ee2e853d2704abe1e3fafaf059b0dd16a

# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

# docker ps -a
CONTAINER ID        IMAGE               COMMAND              CREATED             STATUS                     PORTS               NAMES
1e12902555a7        apache_httpd        "httpd-foreground"   8 seconds ago       Exited (1) 7 seconds ago                       apache_httpd

# docker logs apache_httpd 
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
(13)Permission denied: AH00091: httpd: could not open error log file /proc/self/fd/2.
AH00015: Unable to open logs
Comment 3 Antonio Murdaca 2017-06-09 18:28:05 UTC 
Can you disable selinux and re-try?
Comment 4 Micah Abbott 2017-06-09 19:22:19 UTC 
Yeah, disabling SELinux works around it.
Comment 5 Micah Abbott 2017-06-09 19:27:42 UTC 
Similar problems with a 'httpd' container based on 'docker.io/nginx'

Build with this Dockerfile:

https://github.com/projectatomic/atomic-host-tests/blob/master/tests/docker-build-httpd/files/nginx_httpd/Dockerfile

The logs from that container:

# docker logs nginx_httpd 
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2017/06/09 19:14:34 [emerg] 1#1: open() "/var/log/nginx/error.log" failed (13: Permission denied)


Disabling SELinux in this case also worked around the error.
Comment 6 Daniel Walsh 2017-06-10 12:20:51 UTC 
I believe this is an issue in the latest container-selinux package.


Whenever you report an error with SELinux always attach the AVC messages.

ausearch -m avc -ts recent.



Also give me

ps -eZ | grep docker


I suspect that docker is running with something like
container_runtime_t:s0-s0:c10123

It used to run as
container_runtime_t:s0


I believe this is related to a problem we have seen in Fedora about docker leaking fds into the container to be used for stdin, stdout and stderr. I changed the way that the container runtimes run which is causing a new SELinux issue.
Comment 7 Micah Abbott 2017-06-12 13:42:18 UTC 
There's no `ausearch` on RHELAH, so the best I can give you is some grepped journal entries.


# journalctl -b | grep -e avc -e audit -e denied
Jun 12 13:39:56 micah-rhelah-vm0609a.localdomain kernel: type=1401 audit(1497274796.569:12): op=security_compute_av reason=bounds scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:svirt_lxc_net_t:s0-s0:c0.c1023 tclass=process perms=transition,sigchld,sigstop,signull,signal,getattr
Jun 12 13:39:56 micah-rhelah-vm0609a.localdomain kernel: type=1401 audit(1497274796.603:13): op=security_compute_av reason=bounds scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:svirt_lxc_net_t:s0-s0:c0.c1023 tclass=process perms=transition,sigchld,sigstop,signull,signal,getattr
Jun 12 13:40:29 micah-rhelah-vm0609a.localdomain kernel: type=1400 audit(1497274829.974:14): avc:  denied  { open } for  pid=14526 comm="httpd" path="pipe:[78235]" dev="pipefs" ino=78235 scontext=system_u:system_r:svirt_lxc_net_t:s0:c533,c834 tcontext=system_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=fifo_file
Jun 12 13:40:29 micah-rhelah-vm0609a.localdomain dockerd-current[5726]: (13)Permission denied: AH00091: httpd: could not open error log file /proc/self/fd/2.



# ps -eZ | grep docker
system_u:system_r:container_runtime_t:s0-s0:c0.c1023 5726 ? 00:00:45 dockerd-current
system_u:system_r:container_runtime_t:s0-s0:c0.c1023 5765 ? 00:00:03 docker-containe
Comment 8 Daniel Walsh 2017-06-14 18:41:32 UTC 
grep avc /var/log/audit/audit.log
Comment 9 Micah Abbott 2017-06-14 19:37:49 UTC 
(In reply to Daniel Walsh from comment #8)
> grep avc /var/log/audit/audit.log

That doesn't exist on RHELAH systems.
Comment 10 Daniel Walsh 2017-06-14 19:39:17 UTC 
Yes well you gave me the AVC's anyways.

journalctl -b | grep -i avc

:^)
Comment 11 Micah Abbott 2017-06-14 19:39:31 UTC 
(In reply to Micah Abbott from comment #9)
> (In reply to Daniel Walsh from comment #8)
> > grep avc /var/log/audit/audit.log
> 
> That doesn't exist on RHELAH systems.

Well, I won't make a blanket statement.  But it doesn't exist on the system I reproduced this on.
Comment 16 errata-xmlrpc 2017-06-28 15:41:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1626