Bug 1096816

+++ This bug was initially created as a clone of Bug #1096123 +++

Description of problem:
Between docker-io-0.10.0-2.fc20 and docker-io-0.11.1-1.fc20, the following has started failing:

$ docker run -t centos /usr/sbin/useradd test
useradd: failure while writing changes to /etc/passwd

'centos' is the official CentOS 6 image (0b443ba03958).

The Fedora 20 host has SELinux enforcing, and the same issue occurs when set to permissive.  No AVCs are seen.

Version-Release number of selected component (if applicable):
docker-io-0.11.1-1.fc20.x86_64
kernel-3.14.2-200.fc20.x86_64

How reproducible:
Always

Steps to Reproduce:
1. docker pull centos
2. docker run -t centos /usr/sbin/useradd test

Actual results:
useradd: failure while writing changes to /etc/passwd

Expected results:
no output

Additional info:

On 0.10.0, an strace of useradd shows:

open("/etc/group", O_RDONLY|O_CLOEXEC)  = 11
fstat(11, {st_mode=S_IFREG|0644, st_size=379, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feb5efe5000
read(11, "root:x:0:\nbin:x:1:bin,daemon\ndae"..., 4096) = 379
close(11)                               = 0
munmap(0x7feb5efe5000, 4096)            = 0
fchown(10, 500, 12)                     = 0
fchmod(10, 0660)                        = 0
fsync(10)                               = 0
close(10)                               = 0
fstat(6, {st_mode=S_IFREG|0644, st_size=670, ...}) = 0
gettid()                                = 14
open("/proc/self/task/14/attr/fscreate", O_RDONLY) = 10
read(10, "", 4095)                      = 0
close(10)                               = 0
gettid()                                = 14
open("/proc/self/task/14/attr/fscreate", O_RDWR) = 10
write(10, "system_u:object_r:file_t:s0\0", 28) = 28
close(10)                               = 0
fstat(6, {st_mode=S_IFREG|0644, st_size=670, ...}) = 0
umask(077)                              = 022
open("/etc/passwd-", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 10
umask(022)                              = 077
lseek(6, 0, SEEK_SET)                   = 0
read(6, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 670
fstat(10, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feb5efe5000
read(6, "", 4096)                       = 0
write(10, "root:x:0:0:root:/root:/bin/bash\n"..., 670) = 670

While on 0.11.1, strace shows:

open("/etc/group", O_RDONLY|O_CLOEXEC)  = 10
fstat(10, {st_mode=S_IFREG|0644, st_size=379, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2905a38000
read(10, "root:x:0:\nbin:x:1:bin,daemon\ndae"..., 4096) = 379
close(10)                               = 0
munmap(0x7f2905a38000, 4096)            = 0
fchown(9, 500, 12)                      = 0
fchmod(9, 0660)                         = 0
fsync(9)                                = 0
close(9)                                = 0
fstat(5, {st_mode=S_IFREG|0644, st_size=675, ...}) = 0
gettid()                                = 30
open("/proc/self/task/30/attr/fscreate", O_RDONLY) = 9
read(9, "", 4095)                       = 0
close(9)                                = 0
gettid()                                = 30
open("/proc/self/task/30/attr/fscreate", O_RDWR) = -1 EROFS (Read-only file system)
write(2, "useradd: failure while writing c"..., 54useradd: failure while writing changes to /etc/passwd
) = 54

--- Additional comment from Dominic Cleal on 2014-05-09 07:13:07 EDT ---

useradd is just calling libselinux's setfscreatecon, which is being blocked.  On 0.11.1, this library call fails, on 0.10.0/0.9 it works.

--- Additional comment from Paul Morgan on 2014-05-09 09:55:15 EDT ---

when selinux=disabled, i cannot reproduce the bug.

when selinux is enabled (permissive), the bug is always reproducible.

i reproduced using two docker service configs:

* first, use default systemd unit
* second, add `--selinux-enabled` as described at
  http://blog.docker.io/2014/05/docker-0-11-release-candidate-for-1-0/

override default systemd unit...

# cp /usr/lib/systemd/system/docker.service /etc/systemd/system/

# vim /etc/systemd/system/docker.service
# cat !$ 
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
After=network.target

[Service]
ExecStart=/usr/bin/docker -d --selinux-enabled
Restart=on-failure
LimitNOFILE=1048576
LimitNPROC=1048576

[Install]
WantedBy=multi-user.target


# systemctl restart docker.service 
# systemctl status docker.service 
docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; enabled)
   Active: active (running) since Fri 2014-05-09 09:52:25 EDT; 4s ago
     Docs: http://docs.docker.io
 Main PID: 997 (docker)
   CGroup: /system.slice/docker.service
           └─997 /usr/bin/docker -d --selinux-enabled


other system info:

$ rpm -q selinux-policy-targeted
selinux-policy-targeted-3.12.1-158.fc20.noarch

$ uname -a
Linux f20-01.example.com 3.13.9-200.fc20.x86_64 #1 SMP Fri Apr 4 12:13:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ rpm -q docker-io
docker-io-0.11.1-1.fc20.x86_64

$ docker version
Client version: 0.11.1
Client API version: 1.11
Go version (client): go1.2.1
Git commit (client): fb99f99/0.11.1
Server version: 0.11.1
Server API version: 1.11
Git commit (server): fb99f99/0.11.1
Go version (server): go1.2.1
Last stable version: 0.11.1

$ docker images | grep '^centos'
centos                centos6             0b443ba03958        3 weeks ago         297.6 MB
centos                latest              0b443ba03958        3 weeks ago         297.6 MB
centos                6.4                 539c0211cd76        13 months ago       300.6 MB

--- Additional comment from Lokesh Mandvekar on 2014-05-09 13:41:51 EDT ---

I have a new scratch build http://kojipkgs.fedoraproject.org//work/tasks/2230/6832230/docker-io-0.11.1-3.fc20.x86_64.rpm 


with this build, this error doesn't occur with fedora:20, but still does with centos

dwalsh, comments?

--- Additional comment from Dominic Cleal on 2014-05-09 13:45:57 EDT ---

Comparing the straces between el6 and fedora:20, I don't see any of the same accesses to attr/fscreate on f20 that are in the bug description.  The source of shadow-utils between el6 & f20 looks very different too, I can't see any setfscreatecon calls in the useradd code path.

--- Additional comment from Daniel Walsh on 2014-05-09 16:51:06 EDT ---

The problem is inside the container it sees SELinux as being enabled, which is the bug.

If you do id -Z, does it complain inside the container?

 docker run --rm -t -i fedora sh
sh-4.2# id -Z
id: --context (-Z) works only on an SELinux-enabled kernel
sh-4.2# mount | grep /sys
sysfs on /sys type sysfs (ro,relatime,seclabel)


SELinux sees the container as being disabled since /sys/fs/selinux is mounted as read/only, this will tell useradd NOT to try to do any SELinux stuff while in the container.

--- Additional comment from Fedora Update System on 2014-05-10 00:04:23 EDT ---

docker-io-0.11.1-3.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/docker-io-0.11.1-3.fc20

--- Additional comment from Dominic Cleal on 2014-05-10 08:01:14 EDT ---

(In reply to Daniel Walsh from comment #5)
> The problem is inside the container it sees SELinux as being enabled, which
> is the bug.
> 
> If you do id -Z, does it complain inside the container?

No, it runs and reports a context.

>  docker run --rm -t -i fedora sh
> sh-4.2# id -Z
> id: --context (-Z) works only on an SELinux-enabled kernel
> sh-4.2# mount | grep /sys
> sysfs on /sys type sysfs (ro,relatime,seclabel)

$ rpm -q docker-io
docker-io-0.9.1-1.fc20.x86_64
$ docker run -i -t centos /bin/bash
bash-4.1# id -Z
system_u:system_r:docker_t:s0
bash-4.1# mount | grep sys
sysfs on /sys type sysfs (rw,seclabel,nosuid,nodev,noexec,relatime)

$ rpm -q docker-io
docker-io-0.11.1-3.fc20.x86_64
$ docker run -i -t centos /bin/bash
bash-4.1# id -Z
system_u:system_r:svirt_lxc_net_t:s0:c231,c400
bash-4.1# mount | grep /sys
sysfs on /sys type sysfs (ro,seclabel,relatime)

> SELinux sees the container as being disabled since /sys/fs/selinux is
> mounted as read/only, this will tell useradd NOT to try to do any SELinux
> stuff while in the container.

/sys is correctly read-only as you expected, but it seems useradd's still doing SELinux stuff then.  These packages are installed inside the EL6 container:

libselinux-2.0.94-5.3.el6_4.1.x86_64
libselinux-utils-2.0.94-5.3.el6_4.1.x86_64
shadow-utils-4.1.4.2-13.el6.x86_64

Calling is_selinux_enabled() on Fedora is returning 0, while on EL6 it's returning 1.  Another difference - on Fedora, getenforce returns "Disabled" but on EL6 it prints:

# getenforce
getenforce:  getenforce() failedbash-4.1# 

/selinux exists within the container, but nothing is actually mounted there.  It appears to be simply a directory on the root filesystem (/selinux/booleans exists as an empty dir).  No other SELinux mounts are visible.

Looking at libselinux-2.0.94, I think it's seeing selinuxfs listed in /proc/filesystems and assuming SELinux is enabled because of this.  libselinux-2.2.1 on F20 doesn't seem to have this code.

libselinux-2.0.94/src/enabled.c:
        /* Drop back to detecting it the long way. */
        fp = fopen("/proc/filesystems", "r");
        if (!fp)
                return -1;

        __fsetlocking(fp, FSETLOCKING_BYCALLER);
        while ((num = getline(&buf, &len, fp)) != -1) {
                if (strstr(buf, "selinuxfs")) {
                        enabled = 1;
                        break;
                }
        }

# grep selinux /proc/filesystems 
nodev	selinuxfs

(All the above was tested with docker-io-0.11.1-3.fc20)

--- Additional comment from Fedora Update System on 2014-05-12 01:28:00 EDT ---

Package docker-io-0.11.1-3.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing docker-io-0.11.1-3.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-6281/docker-io-0.11.1-3.fc20
then log in and leave karma (feedback).

--- Additional comment from Dominic Cleal on 2014-05-12 06:34:11 EDT ---

(In reply to Dominic Cleal from comment #7)
> Looking at libselinux-2.0.94, I think it's seeing selinuxfs listed in
> /proc/filesystems and assuming SELinux is enabled because of this. 
> libselinux-2.2.1 on F20 doesn't seem to have this code.

Bug #835146 (against EL6) seems to confirm this, suggesting a backport of the patch that removes the /proc/filesystems based check.