Bug 1440875 - Docker image fails to run when selinux is enforcing on aarch64
Summary: Docker image fails to run when selinux is enforcing on aarch64
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: container-selinux
Version: 26
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Lokesh Mandvekar
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: ARMTracker 1445610
TreeView+ depends on / blocked
 
Reported: 2017-04-10 16:18 UTC by Paul Whalen
Modified: 2018-01-16 15:29 UTC (History)
15 users (show)

(edit)
Clone Of:
: 1445610 (view as bug list)
(edit)
Last Closed: 2018-01-16 15:25:25 UTC


Attachments (Terms of Use)
Audit log (142.10 KB, text/plain)
2017-04-10 16:18 UTC, Paul Whalen
no flags Details
journalctl (803.14 KB, text/plain)
2017-04-10 16:19 UTC, Paul Whalen
no flags Details
docker on aarch64 (5.58 KB, text/plain)
2017-05-30 22:11 UTC, Paul Whalen
no flags Details
selinux content of /usr on ppc64le (1.10 KB, text/plain)
2017-06-02 10:43 UTC, Sinny Kumari
no flags Details
selinux context of /var on ppc64le host vm (1.86 MB, text/plain)
2017-06-02 17:45 UTC, Sinny Kumari
no flags Details
Audit log on ppc64le (318.64 KB, text/plain)
2017-10-06 14:32 UTC, Sinny Kumari
no flags Details

Description Paul Whalen 2017-04-10 16:18:34 UTC
Created attachment 1270528 [details]
Audit log

Description of problem:
Docker image fails to run when selinux is running in enforcing on aarch64.

Version-Release number of selected component (if applicable):
selinux-policy-3.13.1-249.fc26.noarch
docker-1.13.1-6.git5be1549.fc26.aarch64

How reproducible:
Everytime.

Steps to Reproduce:
1. Download and load F26 Alpha docker image for aarch64.
2. docker run -it --rm fedora-docker-base-26_alpha-1.7.aarch64 /bin/bash

Actual results:
Image fails to start, from logs:
Apr 10 11:46:08 mustang.friendly-neighbours.com audit[1147]: AVC avc:  denied  { read execute } for  pid=1147 comm="bash" path="/usr/bin/bash" dev="dm-0" ino=8471581 scontext=system_u:system_r:container_t:s0:c758,c895 tcontext=system_u:o
Apr 10 11:46:08 mustang.friendly-neighbours.com audit[1147]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:container_t:s0:c758,c895 pid=1147 comm="bash" exe="/usr/bin/bash" sig=11 res=1


Expected results:
bash prompt. 

Additional info:
Starting with selinux in permissive works as expected.

Comment 1 Paul Whalen 2017-04-10 16:19 UTC
Created attachment 1270529 [details]
journalctl

Comment 2 Daniel Walsh 2017-04-10 16:43:11 UTC
This looks like you are mounting and attempting to run fusefs content under a container?

Have you turned on the virt_sandbox_use_fusefs boolean?

Comment 3 Paul Whalen 2017-04-10 20:10:42 UTC
No changes were made to the default. This works on other arches (x86, armhfp) without turning on that boolean. 
 
On x86:

sudo docker run -it fedora-docker-base-26_alpha-1.7.x86_64 /bin/bash
[root@0b069afd9787 /]# exit
[pwhalen@i7 ~]$ getenforce 
Enforcing
[pwhalen@i7 ~]$ getsebool virt_sandbox_use_fusefs
virt_sandbox_use_fusefs --> off

On aarch64, when the boolean is turned on it does work as expected:

setsebool virt_sandbox_use_fusefs on
docker run -it --rm fedora-docker-base-26_alpha-1.7.aarch64 /bin/bash
[root@56b0cce69c90 /]# exit

Comment 4 Paul Whalen 2017-05-30 22:10:50 UTC
Still seeing this with:

selinux-policy-3.13.1-254.fc26.noarch
docker-1.13.1-13.git51eb16e.fc26.aarch64

Comment 5 Paul Whalen 2017-05-30 22:11 UTC
Created attachment 1283527 [details]
docker on aarch64

Comment 6 Daniel Walsh 2017-05-31 11:55:51 UTC
I need avc's.

ausearch -m avc -ts recent

Comment 7 Paul Whalen 2017-05-31 19:11:53 UTC
time->Wed May 31 14:52:30 2017
type=AVC msg=audit(1496256750.728:262): avc:  denied  { read execute } for  pid=2078 comm="bash" path="/usr/bin/bash" dev="dm-0" ino=25705300 scontext=system_u:system_r:container_t:s0:c73,c693 tcontext=system_u:object_r:fusefs_t:s0 tclass=file permissive=0

Comment 8 Daniel Walsh 2017-05-31 19:57:04 UTC
setsebool -P virt_sandbox_use_fusefs 1

Should enable these.

Comment 9 Sinny Kumari 2017-06-01 13:27:43 UTC
(In reply to Paul Whalen from comment #0)
> Created attachment 1270528 [details]
> Audit log
> 
> Description of problem:
> Docker image fails to run when selinux is running in enforcing on aarch64.
> 
> Version-Release number of selected component (if applicable):
> selinux-policy-3.13.1-249.fc26.noarch
> docker-1.13.1-6.git5be1549.fc26.aarch64
> 
> How reproducible:
> Everytime.
> 
> Steps to Reproduce:
> 1. Download and load F26 Alpha docker image for aarch64.
> 2. docker run -it --rm fedora-docker-base-26_alpha-1.7.aarch64 /bin/bash

Similar problem on ppc64le as well. Have to do setenforce 0 in order to run container. Tried it on F26 beta 1.4 docker image (https://kojipkgs.fedoraproject.org/compose/26/Fedora-26-20170531.0/compose/Docker/ppc64le/images/)

Comment 10 Daniel Walsh 2017-06-01 13:44:29 UTC
Did turning on this boolean fix the issue?

Comment 11 Sinny Kumari 2017-06-01 14:00:18 UTC
(In reply to Daniel Walsh from comment #10)
> Did turning on this boolean fix the issue?

Yes Dan, works fine after setting virt_sandbox_use_fusefs on

Comment 12 Daniel Walsh 2017-06-01 18:29:20 UTC
Before I close this, are you actually using fusefs?

Comment 13 Sinny Kumari 2017-06-02 07:47:49 UTC
(In reply to Daniel Walsh from comment #12)
> Before I close this, are you actually using fusefs?

I haven't done any modification with base docker image (obtained from https://kojipkgs.fedoraproject.org/compose/26/Fedora-26-20170531.0/compose/Docker/ppc64le/images/ )

After downloading and loading image, I simply ran 
$ sudo docker run -i -t image_name /bin/bash

which didn't succeed.

Ouput of command "ausearch -m avc -ts recent" gives following result though:

time->Fri Jun  2 09:39:06 2017
type=AVC msg=audit(1496389146.567:409): avc:  denied  { read execute } for  pid=2905 comm="bash" path="/usr/bin/bash" dev="dm-0" ino=17010971 scontext=system_u:system_r:container_t:s0:c726,c987 tcontext=system_u:object_r:fusefs_t:s0 tclass=file permissive=0

Later with setting virt_sandbox_use_fusefs ON , docker container runs fine

Comment 14 Daniel Walsh 2017-06-02 10:14:46 UTC
Inside of the container what does

ls -lZ /usr

Show?  Is everything on a fusefs_t file system?

Comment 15 Sinny Kumari 2017-06-02 10:43 UTC
Created attachment 1284395 [details]
selinux content of /usr on ppc64le

No, everything are of type container_file_t

Comment 16 Daniel Walsh 2017-06-02 11:11:28 UTC
The question I have is where are these fuse file system files coming from?

Comment 17 Sinny Kumari 2017-06-02 12:13:56 UTC
I didn't find much useful data to understand from where fuse FS files are coming.

I tried following inside container in case it is useful:
[root@c82177200ad4 /]# ls -lRZ | grep fuse
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c153,c932  9 Jun  1 01:16 sys-fs-fuse-connections.mount -> /dev/null
-rw-r--r--. 1 root root system_u:object_r:container_file_t:s0:c153,c932  754 Mar 16 09:25 sys-fs-fuse-connections.mount
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c153,c932 32 Mar 16 09:25 sys-fs-fuse-connections.mount -> ../sys-fs-fuse-connections.mount

sys-fs-fuse-connections.mount is coming from systemd package.

Note: Didn't find sys-fs-fuse-connections.mount file inside x86_64 docker container (docker image taken from https://kojipkgs.fedoraproject.org/compose/26/Fedora-26-20170531.0/compose/Docker/x86_64/images/)

Also tried, 
[root@c82177200ad4 /]# mount |grep fuse

which has no matching result

Let me know if I should try something else.

Comment 18 Daniel Walsh 2017-06-02 14:38:23 UTC
If you execute that command on the host, do you see much?

Comment 19 Sinny Kumari 2017-06-02 17:45 UTC
Created attachment 1284491 [details]
selinux context of /var on ppc64le host vm

On host machine, selinux context of /var seems interesting (rest of directories under / doesn't have any fusefs_t type). Especially files under directory /var/lib/docker/overlay2/ seem to have fusefs_t FS type.

Comment 20 Daniel Walsh 2017-06-02 18:19:27 UTC
That would seem to be a bug or a setup problem.  Those files should be on the local system...

I think there is a problem with overlayfs on aarch64?

Comment 21 Paul Whalen 2017-10-05 19:49:37 UTC
This is working on aarch64 in the F27 Beta. 

[pwhalen@mustang ~]$ uname -r
4.13.3-300.fc27.aarch64
[pwhalen@mustang ~]$ getenforce
Enforcing
[pwhalen@mustang ~]$ getsebool virt_sandbox_use_fusefs
virt_sandbox_use_fusefs --> off
[pwhalen@mustang ~]$ sudo docker run -it --rm fedora-docker-base-27_beta-1.5.aarch64 bash
[root@9bba18573d12 /]#

Comment 22 Sinny Kumari 2017-10-06 05:16:41 UTC
Problem still exist on ppc64le, F27 Beta.
Was there any changes made to fix this issue?

Comment 23 Daniel Walsh 2017-10-06 13:47:44 UTC
Sinny please attach AVC messages.

Comment 24 Sinny Kumari 2017-10-06 14:32 UTC
Created attachment 1335349 [details]
Audit log on ppc64le

Hi Dan,

AVC message on F27 Beta, ppc64le:

type=AVC msg=audit(1507186032.058:285): avc:  denied  { map } for  pid=1956 comm="bash" path="/usr/bin/bash" dev="dm-0" ino=1338074 scontext=system_u:system_r:container_t:s0:c20,c706 tcontext=system_u:object_r:fusefs_t:s0 tclass=file permissive=0
type=ANOM_ABEND msg=audit(1507186032.059:286): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:container_t:s0:c20,c706 pid=1956 comm="bash" exe="/usr/bin/bash" sig=11 res=1

Complete audit log is available in attachment.

Comment 25 Daniel Walsh 2017-10-06 14:44:18 UTC
Stephen, you say that this is only happening when a file is mmap'd?  Do you think bash memory mapps fusefs files?

Comment 26 Stephen Smalley 2017-10-06 15:36:33 UTC
map permission is only checked during mmap(2) or other calls that internally invoke vm_mmap(), e.g. execve(2) for mapping the executable. bash could easily be invoking mmap(2) directly or via a glibc function for reading a file; the fact that the file happens to live on fusefs is immaterial.
If you enable syscall audit, then we can see the actual system call that triggered the avc denial.

Comment 27 Daniel Walsh 2017-10-06 15:45:31 UTC
But that means we really need to be fairly loose with the map permissions, if random bash scripts can cause breakage.  I believe we should add it to can_exec.

Comment 28 Daniel Walsh 2017-10-06 15:52:26 UTC
https://github.com/fedora-selinux/selinux-policy/pull/203

Should fix this along with an updated container-selinux.

Comment 29 Daniel Walsh 2017-10-06 15:53:04 UTC
Lukas we need this for F27 and F26, and I will rebuild container-selinux off of it.

Comment 30 Lukas Vrabec 2017-10-09 12:31:31 UTC
Dan, 

We have map permission only in F27+. I would say that if you rebuild container-selinux with the latest patches from selinux-policy, it should be fixed.

Comment 31 Daniel Walsh 2017-10-09 14:02:25 UTC
Fixed in container-selinux-2.28.

Comment 32 Sinny Kumari 2017-10-12 08:50:43 UTC
Thanks Dan!
It works now on ppc64le as well.

# getenforce
Enforcing
# getsebool virt_sandbox_use_fusefs
virt_sandbox_use_fusefs --> off
# rpm -qa container-selinux
container-selinux-2.28-1.fc27.noarch
# docker run --rm -it fedora-docker-base-27-20171010.n.0.ppc64le bash
[root@bf98e838a90f /]#

Comment 33 Paul Whalen 2018-01-16 15:25:25 UTC
Closing as this was fixed in F28. Thanks!


Note You need to log in before you can comment on or make changes to this bug.