Bug 1403398

Summary: f25 - docker doesn't label overlay2 directory correctly
Product: [Fedora] Fedora Reporter: Dusty Mabe <dustymabe>
Component: dockerAssignee: Antonio Murdaca <amurdaca>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 25CC: adimania, admiller, amurdaca, anass.1430, dwalsh, ichavero, jcajka, jchaloup, lsm5, marianne, miminar, nalin, riek, vbatts
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: docker-1.12.4-2.git1b5971a.fc25 docker-1.12.4-6.git1b5971a.fc25 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-16 00:27:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dusty Mabe 2016-12-09 23:47:39 UTC
Description of problem:

The right policy is in place for the /var/lib/docker/overlay2/ directory to be labeled appropriately but somehow the files still get created with the wrong container_var_lib_t label. 


```
-bash-4.3# docker run -it --rm busybox ls 
Unable to find image 'busybox:latest' locally
Trying to pull repository docker.io/library/busybox ... 
sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912: Pulling from docker.io/library/busybox
56bec22e3559: Pull complete 
Digest: sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912
Status: Downloaded newer image for docker.io/busybox:latest
panic: standard_init_linux.go:175: exec user process caused "permission denied" [recovered]
        panic: standard_init_linux.go:175: exec user process caused "permission denied"

goroutine 1 [running, locked to thread]:
panic(0x6f2fc0, 0xc42014d260)
        /usr/lib/golang/src/runtime/panic.go:500 +0x1a1
github.com/urfave/cli.HandleAction.func1(0xc42007f748)
        /builddir/build/BUILD/docker-97974ae27b1b4dd0e6a54d1755a07b3703453fac/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x247
panic(0x6f2fc0, 0xc42014d260)
        /usr/lib/golang/src/runtime/panic.go:458 +0x243
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc42007f198, 0xc42001e078, 0xc42007f238)
        /builddir/build/BUILD/docker-97974ae27b1b4dd0e6a54d1755a07b3703453fac/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x18f
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc42004e690, 0xaac9a0, 0xc42014d260)
        /builddir/build/BUILD/docker-97974ae27b1b4dd0e6a54d1755a07b3703453fac/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353
main.glob..func8(0xc420082780, 0x0, 0x0)
        /builddir/build/BUILD/docker-97974ae27b1b4dd0e6a54d1755a07b3703453fac/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/main_unix.go:26 +0x66
reflect.Value.call(0x6ddd40, 0x769d68, 0x13, 0x73c249, 0x4, 0xc42007f708, 0x1, 0x1, 0x4d1798, 0x732080, ...)
        /usr/lib/golang/src/reflect/value.go:434 +0x5c8
reflect.Value.Call(0x6ddd40, 0x769d68, 0x13, 0xc42007f708, 0x1, 0x1, 0xac2700, 0xc42007f6e8, 0x4da776)
        /usr/lib/golang/src/reflect/value.go:302 +0xa4
github.com/urfave/cli.HandleAction(0x6ddd40, 0x769d68, 0xc420082780, 0x0, 0x0)
        /builddir/build/BUILD/docker-97974ae27b1b4dd0e6a54d1755a07b3703453fac/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x1e0
github.com/urfave/cli.Command.Run(0x73c415, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74da56, 0x51, 0x0, ...)
        /builddir/build/BUILD/docker-97974ae27b1b4dd0e6a54d1755a07b3703453fac/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xc3b
github.com/urfave/cli.(*App).Run(0xc4200e0000, 0xc42000c120, 0x2, 0x2, 0x0, 0x0)
        /builddir/build/BUILD/docker-97974ae27b1b4dd0e6a54d1755a07b3703453fac/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x611
main.main()
        /builddir/build/BUILD/docker-97974ae27b1b4dd0e6a54d1755a07b3703453fac/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/main.go:137 +0xbd6
-bash-4.3# ausearch -m avc
----
time->Fri Dec  9 23:22:38 2016
type=PROCTITLE msg=audit(1481325758.644:246): proctitle=2F70726F632F73656C662F65786500696E6974
type=SYSCALL msg=audit(1481325758.644:246): arch=c000003e syscall=59 success=no exit=-13 a0=c420119c40 a1=c420119c50 a2=c42014e330 a3=0 items=0 ppid=1579 pid=1593 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=4294967295 comm="exe" exe="/usr/libexec/docker/docker-runc-current" subj=system_u:system_r:container_runtime_t:s0 key=(null)
type=AVC msg=audit(1481325758.644:246): avc:  denied  { entrypoint } for  pid=1593 comm="exe" path="/bin/ls" dev="overlay" ino=24899 scontext=system_u:system_r:container_t:s0:c280,c907 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0

```

A restorecon clears things up and any future docker run commands even with newer images work just fine. 




Version-Release number of selected component (if applicable):
```
-bash-4.3# rpm -q docker 
docker-1.12.3-12.git97974ae.fc25.x86_64
-bash-4.3# rpm-ostree status
State: idle
Deployments:
● fedora-atomic:fedora-atomic/25/x86_64/docker-host
       Version: 25.9 (2016-12-07 05:09:55)
        Commit: 7c0b62b1c6226c0826383e899b4f33992445f6a349018685b52e1c10e0d9160e
        OSName: fedora-atomic
-bash-4.3# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.3
Storage Driver: overlay2
 Backing Filesystem: xfs
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Security Options: seccomp selinux
Kernel Version: 4.8.11-300.fc25.x86_64
Operating System: Fedora 25 (Atomic Host)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 2
Total Memory: 3.859 GiB
Name: cloudhost.localdomain
ID: 44LV:SXCN:GHYW:ZJQN:M6XM:XN42:4AFE:7MPR:WWV2:7XYQ:QH6H:TT5E
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
Registries: docker.io (secure)
```


How reproducible:
always


Steps to Reproduce:
1. Start cloud image with the following user-data: 
bootcmd:                                                                                      
 - echo 'ROOT_SIZE=100%FREE' >>  /etc/sysconfig/docker-storage-setup
 - echo 'STORAGE_DRIVER=overlay2' >>  /etc/sysconfig/docker-storage-setup
2. try to run a container `docker run -it --rm busybox ls`
3. observe error

Comment 1 Daniel Walsh 2016-12-10 12:10:36 UTC
Lokesh or Antonio can  you update the version of container-selinux that we are shipping with docker package to the latest in master.


We have policy that says

sesearch -T -s container_runtime_t  | grep overlay
type_transition container_runtime_t container_var_lib_t : dir container_share_t "overlay2"; 
type_transition container_runtime_t container_var_lib_t : dir container_share_t "overlay"; 


These type transitions say when the label type that the docker/runc is running as "container_runtime_t" creates a directory named overlay or overlay2 in a directory labeled container_var_lib_t (The label of /var/lib/docker) it will get created with container_share_t.

After this policy is installed you can verify this happened by removing all content under /var/lib/docker

atomic storage reset

Will do this for you.

Then start docker again and run an overlay based container.

Comment 2 Antonio Murdaca 2016-12-10 13:23:33 UTC
I'll fix this in Fedora (F25 and F26)

Comment 3 Fedora Update System 2016-12-10 17:35:41 UTC
docker-1.12.3-15.git0423d89.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-37c2c59240

Comment 4 Dusty Mabe 2016-12-11 00:59:07 UTC
Upgrading to the newer RPM seems to give errors:

```
[root@localhost ~]# rpm -q docker docker-common container-selinux
docker-1.12.3-12.git97974ae.fc25.x86_64
docker-common-1.12.3-12.git97974ae.fc25.x86_64
container-selinux-1.12.3-12.git97974ae.fc25.x86_64
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# rpm -Uvh https://kojipkgs.fedoraproject.org//packages/docker/1.12.3/15.git0423d89.fc25/x86_64/container-selinux-1.12.3-15.git0423d89.fc25.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/docker/1.12.3/15.git0423d89.fc25/x86_64/docker-1.12.3-15.git0423d89.fc25.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/docker/1.12.3/15.git0423d89.fc25/x86_64/docker-common-1.12.3-15.git0423d89.fc25.x86_64.rpm 
Retrieving https://kojipkgs.fedoraproject.org//packages/docker/1.12.3/15.git0423d89.fc25/x86_64/container-selinux-1.12.3-15.git0423d89.fc25.x86_64.rpm
Retrieving https://kojipkgs.fedoraproject.org//packages/docker/1.12.3/15.git0423d89.fc25/x86_64/docker-1.12.3-15.git0423d89.fc25.x86_64.rpm
Retrieving https://kojipkgs.fedoraproject.org//packages/docker/1.12.3/15.git0423d89.fc25/x86_64/docker-common-1.12.3-15.git0423d89.fc25.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:docker-common-2:1.12.3-15.git0423################################# [ 17%]
   2:container-selinux-2:1.12.3-15.git################################# [ 33%]
/var/lib/selinux/final/targeted/contexts/files/file_contexts: Multiple different specifications for /var/lib/containers(/.*)?  (system_u:object_r:gear_var_lib_t:s0 and system_u:object_r:container_var_lib_t:s0).
/var/lib/selinux/final/targeted/contexts/files/file_contexts: Invalid argument
libsemanage.semanage_validate_and_compile_fcontexts: setfiles returned error code 1.
/usr/sbin/semodule:  Failed!
   3:docker-2:1.12.3-15.git0423d89.fc2################################# [ 50%]
Cleaning up / removing...
   4:docker-2:1.12.3-12.git97974ae.fc2################################# [ 67%]
   5:container-selinux-2:1.12.3-12.git################################# [ 83%]
   6:docker-common-2:1.12.3-12.git9797################################# [100%]
```

Comment 5 Dusty Mabe 2016-12-11 01:04:40 UTC
same thing happens on a pure install: 

  Installing  : container-selinux-2:1.12.3-15.git0423d89.fc25.x86_64                                                                                                                                                                                                     10/21 
/var/lib/selinux/final/targeted/contexts/files/file_contexts: Multiple different specifications for /var/lib/containers(/.*)?  (system_u:object_r:gear_var_lib_t:s0 and system_u:object_r:container_var_lib_t:s0).
/var/lib/selinux/final/targeted/contexts/files/file_contexts: Invalid argument
libsemanage.semanage_validate_and_compile_fcontexts: setfiles returned error code 1.
/usr/sbin/semodule:  Failed!

Comment 6 Anass Ahmed 2016-12-11 01:11:25 UTC
(In reply to Dusty Mabe from comment #5)
> same thing happens on a pure install: 
> 
>   Installing  : container-selinux-2:1.12.3-15.git0423d89.fc25.x86_64        
> 10/21 
> /var/lib/selinux/final/targeted/contexts/files/file_contexts: Multiple
> different specifications for /var/lib/containers(/.*)? 
> (system_u:object_r:gear_var_lib_t:s0 and
> system_u:object_r:container_var_lib_t:s0).
> /var/lib/selinux/final/targeted/contexts/files/file_contexts: Invalid
> argument
> libsemanage.semanage_validate_and_compile_fcontexts: setfiles returned error
> code 1.
> /usr/sbin/semodule:  Failed!

Same here, didn't notice the first time when I upgraded.

Comment 7 Fedora Update System 2016-12-11 03:30:20 UTC
docker-1.12.3-15.git0423d89.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-37c2c59240

Comment 8 Daniel Walsh 2016-12-11 12:37:29 UTC
Do

semodule -d gear
dnf reinstall container-selinux


And it should work.

We need to get gear dropped from selinux-policy.

Comment 9 Fedora Update System 2016-12-13 15:03:55 UTC
docker-1.12.4-2.git1b5971a.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-bb5ee53c0a

Comment 10 Fedora Update System 2016-12-13 21:47:04 UTC
docker-1.12.4-5.git1b5971a.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-2a18b9e056

Comment 11 Fedora Update System 2016-12-14 02:25:08 UTC
docker-1.12.4-2.git1b5971a.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-bb5ee53c0a

Comment 12 Fedora Update System 2016-12-14 15:32:46 UTC
docker-1.12.4-6.git1b5971a.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-44ed3dd527

Comment 13 Fedora Update System 2016-12-14 21:30:45 UTC
docker-1.12.4-2.git1b5971a.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 14 Dusty Mabe 2016-12-14 22:06:17 UTC
docker-1.12.4-6.git1b5971a.fc25 is what fixes this bug, not 1.12.4-2

Comment 15 Fedora Update System 2016-12-15 05:08:33 UTC
docker-1.12.4-6.git1b5971a.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-44ed3dd527

Comment 16 Fedora Update System 2016-12-16 00:27:34 UTC
docker-1.12.4-6.git1b5971a.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.