Are you using the docker package from upstream?  I think upstream docker has a leaking fifo_file into a container?

Yes it is:

docker.io/nginx     latest              958a7ae9e569        2 hours ago         109.4 MB

I've tried with both the btrfs backend and devicemapper backend (only because I just recently switched from devicemapper to test btrfs and thought it may be caused by that but wasn't) - this is probably extra info


audit2why output: 

type=AVC msg=audit(1496171664.557:117030): avc:  denied  { open } for  pid=28813 comm="nginx" path="pipe:[49061950]" dev="pipefs" ino=49061950 scontext=system_u:system_r:container_t:s0:c549,c923 tcontext=system_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=fifo_file permissive=0
        Was caused by:

#Constraint rule:

#       mlsconstrain fifo_file { open } ((h1 dom h2 -Fail-)  or (t1 != mcs_constrained_type -Fail-) ); Constraint DENIED

#       Possible cause is the source level (s0:c549,c923) and target level (s0-s0:c0.c1023) are different.



The exact command used was:

# sudo docker run -p 80:80 nginx


Which produces the following message (when not run as a daemon):

nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2017/05/30 19:18:15 [emerg] 1#1: open() "/var/log/nginx/error.log" failed (13: Permission denied)


When running interactively+tty (with -it), i.e. `docker run -it -p 80:80 nginx` it sends the logs to stdout, which doesn't cause the container to exit due to an AVC. So, I'm guessing this has something to do with the way it's logging. 

I'm unsure of how to proceed or whather this is a proper bug at all?

There is an upstream issue on this.  Basically we are trying to figure out what is going on here.  There seems to be a some kind of fifo_file being passed into the container from the docker/containerd which the container process is then opening.  But the AVC makes little sense, since fifo_files are not supposed to be stored/labelled the way they are.

But since this is not a Fedora Package, the issue should be dealt with upstream.  I would recommend you use the docker package that comes with the distribution.

Oh, sorry, I thought you meant container version.

I am using everything purely from fedora. 


$ docker --version
Docker version 1.12.6, build ae7d637/1.12.6

$ rpm -qa | grep ^docker
docker-1.12.6-6.gitae7d637.fc25.x86_64
docker-common-1.12.6-6.gitae7d637.fc25.x86_64
docker-compose-1.9.0-3.fc25.noarch

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.6
Storage Driver: btrfs
 Build Version: Btrfs v4.6.1
 Library Version: 101
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: null bridge host overlay
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Security Options: seccomp selinux
Kernel Version: 4.10.16-200.fc25.x86_64
Operating System: Fedora 25 (Twenty Five)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 4
Total Memory: 11.66 GiB
Name: icarus.net.overtmind.com
ID: 4FJV:URQK:GQZE:UPFT:YA5I:U5TR:WBIA:4YUA:LHYG:H5IK:LPWC:HNBV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: xenithorb
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
Registries: docker.io (secure)

Can we get path and system call audit info for the denial?
How to do so:

If you edit /etc/audit/rules.d/audit.rules (not to be confused with /etc/audit/audit.rules), and comment out the last line:
#-a task,never

Then add a watch or filter to turn on pathname collection:
echo "-w /etc/shadow -p w" > /etc/audit/rules.d/shadow.rules

Then you can run:
service auditd restart

And it should regenerate /etc/audit/audit.rules and reload that.
Then trigger the denial, and you don't need to include everything from ausearch -m avc; you can be more selective, e.g.
ausearch -i -m avc -ts recent # recent denials, or
ausearch -i -m avc -c nginx # denials with nginx as the comm
or other variants.

Make sure you use -i (interpret) to ausearch.

You should then get a series of records, including a type=PROCTITLE, type=PATH, type=CWD, type=SYSCALL, and type=AVC for each denial.

To restore to your original state afterward, you can rm /etc/audit/rules.d/shadow.rules, uncomment the last line of /etc/audit/rules.d/audit.rules, and service auditd restart again.

----
type=PROCTITLE msg=audit(05/30/2017 16:12:53.511:118276) : proctitle=nginx -g daemon off;
type=PATH msg=audit(05/30/2017 16:12:53.511:118276) : item=4 name=/var/log/nginx/error.log inode=49268639 dev=00:0a mode=fifo,600 ouid=root ogid=root rdev=00:00 obj=system_u:system_r:container_runtime_t:s0-s0:c0.c1023 nametype=NORMAL
type=PATH msg=audit(05/30/2017 16:12:53.511:118276) : item=3 name=/var/log/nginx/error.log inode=49268728 dev=00:4d mode=link,300 ouid=root ogid=root rdev=00:00 obj=system_u:system_r:container_t:s0:c192,c983 nametype=NORMAL
type=PATH msg=audit(05/30/2017 16:12:53.511:118276) : item=2 name=/var/log/nginx/error.log inode=49266539 dev=00:4e mode=link,777 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:container_file_t:s0:c192,c983 nametype=NORMAL
type=PATH msg=audit(05/30/2017 16:12:53.511:118276) : item=1 name=/var/log/nginx/error.log inode=5600 dev=00:27 mode=link,777 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:container_file_t:s0:c192,c983 nametype=NORMAL
type=PATH msg=audit(05/30/2017 16:12:53.511:118276) : item=0 name=/var/log/nginx/ inode=49268725 dev=00:4d mode=dir,500 ouid=root ogid=root rdev=00:00 obj=system_u:system_r:container_t:s0:c192,c983 nametype=PARENT
type=CWD msg=audit(05/30/2017 16:12:53.511:118276) : cwd=/
type=SYSCALL msg=audit(05/30/2017 16:12:53.511:118276) : arch=x86_64 syscall=open success=yes exit=3 a0=0x55cf57ac15ea a1=O_WRONLY|O_CREAT|O_APPEND a2=0644 a3=0x55cf57d10aba items=5 ppid=8435 pid=8451 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=nginx exe=/usr/sbin/nginx subj=system_u:system_r:container_t:s0:c192,c983 key=(null)
type=AVC msg=audit(05/30/2017 16:12:53.511:118276) : avc:  denied  { open } for  pid=8451 comm=nginx path=pipe:[49268639] dev="pipefs" ino=49268639 scontext=system_u:system_r:container_t:s0:c192,c983 tcontext=system_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=fifo_file permissive=1

Ok, so /var/log/nginx/error.log is a symlink to /dev/stderr, which is a symlink to /proc/self/fd/2.  In the case of your container, that's a reference to a pipe provided by docker, I assume.  And nginx is calling open() on that.

Options:
1) docker could relabel the pipe before giving it to the container (fsetfilecon on the pipe fd), or
2) policy could allow it.

I think it would be best if the fifo_file was relabeled.  Weird that we are just beginning to see these now.

Antonio and Mrunal do you know what is going on here?  Is docker setting up a fifo file to listen for stderr?

Yes, docker creates fifos for stdout, stderr and stdin.

And the fifo files are in its /proc/self area?

They are duped to /proc/self/fd/{0,1,2}.

Some further info maybe worth noting:

1. This does NOT occur on any of the current atomic host platforms, CentOS or Fedora 
2. This does occur on a newly net-installed Fedora 25 host (up-to-date)
3. This does occur on a newly net-installed Fedora 25 host with updates-testing (up-to-date) to include one with the most recent `container-selinux-2.16-1.fc25.noarch` patches:

* Wed May 31 2017 Dan Walsh <dwalsh> - 2.15-1
- Allow container types to read/write container_runtime fifo files
- Allow a container runtime to mount on top of its own /proc

(and newer but this one seemed relevant) 

I did a autorelabel for good measure and as far as I can tell it still doesn't work out-of-the-box?

The `docker run nginx` replication command is still valid, of course.

I did notice that this ticket wasn't scheduled as a fix on the bodhi update (https://bodhi.fedoraproject.org/updates/FEDORA-2017-4ba099329f) so no worries, I just wanted to report back with current findings. 

---- 

Off topic but I'm not really sure how to work around this temporarily without setting permissive? Generating a policy the normal way (i.e. with audit2allow) doesn't suffice because I assume this doesn't work with MLS the same way.

Are you saying container-selinux-2.16 does not fix this issue?  If yes, please attach the current AVC.s

$ audit2why << EOF
>  avc:  denied  { open } for  pid=8451 comm=nginx path=pipe:[49268639] dev="pipefs" ino=49268639 scontext=system_u:system_r:container_t:s0:c192,c983 tcontext=system_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=fifo_file permissive=1
> EOF
 avc:  denied  { open } for  pid=8451 comm=nginx path=pipe:[49268639] dev="pipefs" ino=49268639 scontext=system_u:system_r:container_t:s0:c192,c983 tcontext=system_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=fifo_file permissive=1

	Was caused by:
()
#Constraint rule:

#	mlsconstrain fifo_file { open } ((h1 dom h2 -Fail-)  or (t1 != mcs_constrained_type -Fail-) ); Constraint DENIED

#	Possible cause is the source level (s0:c192,c983) and target level (s0-s0:c0.c1023) are different.

Dan, you can't allow this in policy without either a) removing container_t from mcs_constrained_type, which would then disable MCS-based container isolation, or b) changing the constraint itself to exempt e.g. container_runtime_t.

For the latter, you would do something like:
mlsconstrain fifo_file { open } ((h1 dom h2) or (t1 != mcs_constrained_type) or (t2 == container_runtime_t));

More generally, you would have a mcstrustedobject or similar and use that above, then add container_runtime_t to it.  MLS constraints have something along those lines.

(In reply to Daniel Walsh from comment #15)
> Are you saying container-selinux-2.16 does not fix this issue?  If yes,
> please attach the current AVC.s

They appear to have remained the same, I also discovered that the httpd container suffers from the same exact problem: https://bugzilla.redhat.com/show_bug.cgi?id=1458590

$ sudo cat /var/log/audit/audit.log | grep AVC | tail -n2
type=AVC msg=audit(1496678011.717:27191): avc:  denied  { open } for  pid=2670 comm="nginx" path="pipe:[13115525]" dev="pipefs" ino=13115525 scontext=system_u:system_r:container_t:s0:c121,c246 tcontext=system_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=fifo_file permissive=0
type=AVC msg=audit(1496678011.724:27192): avc:  denied  { open } for  pid=2670 comm="nginx" path="pipe:[13115525]" dev="pipefs" ino=13115525 scontext=system_u:system_r:container_t:s0:c121,c246 tcontext=system_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=fifo_file permissive=0

$ rpm -q container-selinux
container-selinux-2.16-1.fc25.noarch

For context, I have rebooted after upgrading container-selinux with and without autorelabeling, and replicated on newly installed Fedora 25 installations (KVM VMs), as well as on my bare metal machine.

This cannot be fixed via container-selinux. Requires a change to selinux-policy-targeted.

Yes I think the best and most secure manner is to modify docker/containerd to label the fifo to match the label of the containers.

container_file_t:MCS Label.

Which is what we need. Simple fix will not work.

Antonio can you find where these fifo_files are created?  We will need to change their contexts so that the containers can write to them.

I'm curious as to why this does not occur in Fedora 25 AH? Is that worth exploring?

To be clear, you are looking for the pipe used as the stdin/stdout/stderr for the container.  Then you want to fsetfilecon() it to the container process label.  I don't think setfscreatecon() before pipe() will work since we don't implement support for labeling new pipe inodes with a specified context at creation time, so you'd have to fsetfilecon() it after pipe().  May require adding a fsetfilecon() binding for golang.  Also would require allowing container_runtime_t to relabel the pipe/fifo_file from container_runtime_t to container_t.

You can workaround it via a modified constraint as above, but this does permit containers to open other /proc/pid/fd files that refer to pipes in the docker process.

Does Atomic use a different policy?

No, everyone uses container-selinux, with standard Distro policies.

fsetfilecon already exists for Golang.

The current tree for FAH 25 is:

  fedora-atomic:fedora-atomic/25/x86_64/docker-host
             Version: 25.127 (2017-05-19 21:57:03)
              Commit: cdd359911de49f3a8199ffd41a9894019562001d6cf9be66e1894c31b6fa1c66


which has:

selinux-policy-targeted-3.13.1-225.16.fc25.noarch
container-selinux-2.10-1.fc25.noarch


container-selinux differs from Fedora 25 (updates-testing enabled) by a few versions:

selinux-policy-targeted-3.13.1-225.16.fc25.noarch
container-selinux-2.16-1.fc25.noarch

---

FAH seems to be using LVM thin provisioning by default for docker storage, whereas what triggered this on my other machines was both with docker's btrfs storage backend (my primary) and also tested with the devicemapper loopback strategy. (Not blaming that per se, just that those are the differences I could see myself)

$ seinfo -a mcs_constrained_type -x
Type Attributes: 1
   attribute mcs_constrained_type;
	container_t
	netlabel_peer_t
	openshift_app_t
	openshift_t
	sandbox_min_t
	sandbox_net_t
	sandbox_web_t
	sandbox_x_t
	svirt_kvm_net_t
	svirt_qemu_net_t
	svirt_t
	svirt_tcg_t

Do you get the same output on AH?

Nope. I copy and pasted that on two default AH systems and the node that I'm having issues with and they all returned:

[root@node2 ~]# seinfo -a mcs_constrained_type -x
ERROR: Unable to open policy mcs_constrained_type.
ERROR: No such file or directory

So while they all returned that error, there are no differences between the normal fedora system (where the issues occur) and the AH systems (where they don't) 

I'll admit I'm outside of my selinux element at the moment...

Sorry, I wrongly used setools4 syntax.  You want it without a space between -a and mcs_constrained_type:
$ seinfo -amcs_constrained_type -x
   mcs_constrained_type
      netlabel_peer_t
      openshift_t
      openshift_app_t
      sandbox_min_t
      sandbox_x_t
      sandbox_web_t
      sandbox_net_t
      test_overlay_client_t
      svirt_t
      svirt_tcg_t
      container_t
      svirt_qemu_net_t
      svirt_kvm_net_t

and the important thing is whether container_t is in that list.

FAH 25:

$ seinfo -amcs_constrained_type -x
   mcs_constrained_type
      netlabel_peer_t
      openshift_t
      openshift_app_t
      sandbox_min_t
      sandbox_x_t
      sandbox_web_t
      sandbox_net_t
      svirt_t
      svirt_tcg_t
      container_t
      svirt_qemu_net_t
      svirt_kvm_net_t

Fedora 25: 

$ seinfo -amcs_constrained_type -x
   mcs_constrained_type
      netlabel_peer_t
      openshift_t
      openshift_app_t
      sandbox_min_t
      sandbox_x_t
      sandbox_web_t
      sandbox_net_t
      svirt_t
      svirt_tcg_t
      container_t
      svirt_qemu_net_t
      svirt_kvm_net_t

They appear to be the same.

Are your container processes running in container_t?
ps -eZ | grep nginx

Is docker running in container_runtime_t?
ps -eZ | grep docker

Is your system enforcing?
getenforce

Is container_t enforcing?
seinfo --permissive

Is there some docker config option that alters this stdin/stdout/stderr pipe approach?

Created attachment 1285130 [details]
docker_info_diff

(In reply to Stephen Smalley from comment #33)
> Are your container processes running in container_t?
> ps -eZ | grep nginx
> 
> Is docker running in container_runtime_t?
> ps -eZ | grep docker
> 
> Is your system enforcing?
> getenforce
> 
> Is container_t enforcing?
> seinfo --permissive

FAH 25 (Enforcing): 

[root@node1 ~]# ps -eZ | grep 'docker\|nginx'
system_u:system_r:container_runtime_t:s0 994 ? 00:00:02 docker-containe
system_u:system_r:container_runtime_t:s0 997 ? 00:14:01 dockerd-current
system_u:system_r:container_runtime_t:s0 5905 ? 00:00:00 docker-containe
system_u:system_r:container_runtime_t:s0 6671 ? 00:00:00 docker-containe
system_u:system_r:container_runtime_t:s0 7228 ? 00:00:00 docker-containe
system_u:system_r:container_runtime_t:s0 8129 ? 00:00:00 docker-containe
system_u:system_r:container_t:s0:c729,c970 8146 ? 00:00:00 nginx
system_u:system_r:container_t:s0:c729,c970 8181 ? 00:00:00 nginx

[root@node1 ~]# seinfo --permissive

Permissive Types: 1
   tlp_t



Fedora 25 (Permissive):

# setenforce 0
# docker run -d nginx 
# ps -eZ | grep 'docker\|nginx'
system_u:system_r:container_runtime_t:s0-s0:c0.c1023 1485 ? 00:00:10 docker-containe
system_u:system_r:container_runtime_t:s0-s0:c0.c1023 14335 ? 00:01:07 dockerd-current
system_u:system_r:container_runtime_t:s0-s0:c0.c1023 30296 ? 00:00:00 docker-containe
system_u:system_r:container_t:s0:c572,c802 30316 ? 00:00:00 nginx
system_u:system_r:container_t:s0:c572,c802 30367 ? 00:00:00 nginx

# seinfo --permissive

Permissive Types: 1
   tlp_t

(Fedora 25 container was run in permissive mode)

> Is there some docker config option that alters this stdin/stdout/stderr pipe
> approach?

I'm not really sure. I did a side-by-side diff of `docker info` and the main differences were that the kernel versions differed:
   
Kernel Version: 4.11.3-200.fc25.x86_64'
Operating System: Fedora 25 (Twenty Five)

Kernel Version: 4.10.15-200.fc25.x86_64
Operating System: Fedora 25 (Atomic Host)

and (after switching back the plain Fedora node back to devicemapper to mitigate differences): 

Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata

Was present on the plain Fedora 25 node. It seems to default to using a loop device mounted XFS formatted file for the LVM pools if there is no raw block device to do it (as they are present by default in AH)

(diff to see for yourself attached above)

I reproduced locally.
Dan, the difference appears to be that docker is running at s0 on Fedora 25 AH and fully ranged on Fedora 25.  This seems to have been a change in container-selinux, a68dc2fbee7a11a6cd617020eded032aa4fc12a5.  Was that intentional?

The offending lines in container-selinux container.te are:
ifdef(`enable_mcs',`
       init_ranged_daemon_domain(container_runtime_t, container_runtime_exec_t, s0 - mcs_systemhigh)
')

Previously it was just running at s0 and therefore accessible by container_t as long as it was allowed by TE rules.

Right, I added these lines to be able to run containers on an MLS system.  I probably could have left the mcs stuff alone and will revert until we fix docker.  Bottom line is fixing the labels will allow us to tighten the security.

Building container-selinux-2.17

I have verified that container-selinux-2.17-1.fc25 (FEDORA-2017-483d4b64eb) temporarily fixes this issue

This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Fedora 25 changed to end-of-life (EOL) status on 2017-12-12. Fedora 25 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.