Bug 1214810

Summary: SELinux blocking k8s applications
Product: [Retired] Atomic Reporter: Jay Vyas <jvyas>
Component: kernelAssignee: Daniel Walsh <dwalsh>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: eparis, jzb, mkovacik, pmorie, tstclair, walters
Target Milestone: ---Keywords: EC2
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1165982
: 1222060 1227096 (view as bug list) Environment:
Last Closed: 2016-06-10 13:41:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1165982, 1193590    
Bug Blocks: 1222060, 1227096    

Description Jay Vyas 2015-04-23 15:07:03 UTC
Description of the problem

In atomic with kubernetes, running the examples/k8petstore application, we get the following trace.

host12-rack10: Apr 23 10:55:47 host12-rack10.scale.openstack.engineering.redhat.com kernel: type=1400 audit(1429800947.568:5672): avc:  denied  { read } for  pid=6679 comm="redis-server" name="dump.rdb" dev="dm-1" ino=12757262 scontext=system_u:system_r:svirt_lxc_net_t:s0:c61,c652 tcontext=system_u:object_r:docker_var_lib_t:s0 tclass=file
host12-rack10: Apr 23 10:55:47 host12-rack10.scale.openstack.engineering.redhat.com kernel: type=1400 audit(1429800947.566:5671): avc:  denied  { rename } for  pid=6679 comm="redis-server" name="temp-1429800947.9.rdb" dev="dm-1" ino=12757262 scontext=system_u:system_r:svirt_lxc_net_t:s0:c61,c652 tcontext=system_u:object_r:docker_var_lib_t:s0 tclass=file
host12-rack10: Apr 23 10:55:47 host12-rack10.scale.openstack.engineering.redhat.com kernel: type=1400 audit(1429800947.513:5670): avc:  denied  { write open } for  pid=6679 comm="redis-server" path="/data/temp-1429800947.9.rdb" dev="dm-1" ino=12757262 scontext=system_u:system_r:svirt_lxc_net_t:s0:c61,c652 tcontext=system_u:object_r:docker_var_lib_t:s0 tclass=file
host12-rack10: Apr 23 10:55:47 host12-rack10.scale.openstack.engineering.redhat.com kernel: type=1400 audit(1429800947.511:5669): avc:  denied  { create } for  pid=6679 comm="redis-server" name="temp-1429800947.9.rdb" scontext=system_u:system_r:svirt_lxc_net_t:s0:c61,c652 tcontext=system_u:object_r:docker_var_lib_t:s0 tclass=file


This is indicating that SELinux doesn't like the some of the actions being done by the redis components.  

To launch this applicatoin in kubernetes, you can run https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes/master/examples/k8petstore/k8petstore.sh .

Comment 1 Daniel Walsh 2015-04-23 15:17:43 UTC
Is this /data being volume mounted into the container, could it be done with the :Z or :z option.

:Z will tell it to use private labeling, :z Will tell it to use shared labeling. (Shared between containers).

Comment 2 Jay Vyas 2015-05-13 21:08:57 UTC
Its been a while, but This happens on dirs that are not being mounted also iirc.

Comment 3 Jay Vyas 2015-05-14 20:10:46 UTC
FYI, from IRC

<SteveWatt> </html>
<SteveWatt>  the mounted directory is /test
<SteveWatt> it is mounting /opt
<j_brb> SteveWatt, 
<j_brb> May 13 17:29:28 host15-rack11.scale.openstack.engineering.redhat.com kernel: type=1400 audit(1431552568.781:6): avc:  denied  { read } for  pid=122076 comm="nginx" name="hello.html" dev="dm-0" ino=8435109 scontext=system_u:system_r:svirt_lxc_net_t:s0:c125,c316 tcontext=unconfined_u:object_r:var_t:s0 tclass=file
<j_brb> -bash-4.2# 
<j_brb> Theres your error ^^^^^^^^^^^6
<j_brb> journalctl --since yesterday |grep avc <--- that is how to grab the selinux log for it.

Comment 4 Daniel Walsh 2015-05-14 20:23:00 UTC
You need to fix the labels on the volume you are mounting into the container.

Comment 5 Timothy St. Clair 2015-05-14 20:44:13 UTC
@pmorie, any insight here?  

Currently the k8's submission is declarative, we only specify hostPath the actual mounting details are handled by the kubelet.

Comment 6 Jay Vyas 2015-05-15 16:01:04 UTC
More details...  another labelling issue most likely when we attempt to run e2e hostDir test (pending into kubernetes), we get ....  

  Expected error:
      <*errors.errorString | 0xc20825aef0>: {
          s: "pod pod-b4623a77-fb1a-11e4-8c06-ecf4bbc72674 terminated with failure: &{ExitCode:1 Signal:0 Reason: Message: StartedAt:2015-05-15 11:54:45 -0400 EDT FinishedAt:2015-05-15 11:54:45 -0400 EDT Contain
erID:docker://bee5e651dd1ecd1346e53afe8bd293555dd2af55ca174caea6b3e860b4d72df4}",
      }
      pod pod-b4623a77-fb1a-11e4-8c06-ecf4bbc72674 terminated with failure: &{ExitCode:1 Signal:0 Reason: Message: StartedAt:2015-05-15 11:54:45 -0400 EDT FinishedAt:2015-05-15 11:54:45 -0400 EDT ContainerID:doc
ker://bee5e651dd1ecd1346e53afe8bd293555dd2af55ca174caea6b3e860b4d72df4}
  not to have occurred

when attempting to read/write onto a mounted volume.

This is reproduced by 

(1) enabling selinux
(2) running test/e2e/hostDir.go (might be moved, pr is https://github.com/GoogleCloudPlatform/kubernetes/pull/7756 ).

Comment 7 Daniel Walsh 2015-05-15 17:43:41 UTC
Can you pass a Read/Only flag?  If yes then you should be able to pass a Shared/Private flag for relabeling. Of course this has not made it into docker upstream, but it is very close.

Comment 8 Paul Morie 2015-05-19 18:27:55 UTC
Cross-posted from 1222060:

TL;DR recap of my phone convo w/ Tim SC:

There is no treatment for selinux wrt hostDir plugin.  Eventually, the security work going on now will ensure that the selinux context is relabeled so that it plays nice with the selinux context a container is running in.

For now, I would look at whether the policy changed around /var/lib/kubelet during 7.1.1 -> present.

Comment 9 Daniel Walsh 2016-06-10 13:41:13 UTC
I am closing this since I believe it is fixed in current release.