Bug 1096816 - useradd within EL6 container fails: failure while writing changes to /etc/passwd
Summary: useradd within EL6 container fails: failure while writing changes to /etc/passwd
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libselinux
Version: 6.5
Hardware: All
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Miroslav Grepl
QA Contact: Eduard Benes
URL:
Whiteboard:
Depends On: 1096123
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-05-12 13:27 UTC by Daniel Walsh
Modified: 2022-02-21 17:14 UTC (History)
17 users (show)

Fixed In Version: libselinux-2.0.94-5.6.el6
Doc Type: Bug Fix
Doc Text:
Clone Of: 1096123
Environment:
Last Closed: 2014-10-14 06:41:24 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1469 0 normal SHIPPED_LIVE libselinux bug fix update 2014-10-14 01:10:46 UTC

Description Daniel Walsh 2014-05-12 13:27:29 UTC
+++ This bug was initially created as a clone of Bug #1096123 +++

Description of problem:
Between docker-io-0.10.0-2.fc20 and docker-io-0.11.1-1.fc20, the following has started failing:

$ docker run -t centos /usr/sbin/useradd test
useradd: failure while writing changes to /etc/passwd

'centos' is the official CentOS 6 image (0b443ba03958).

The Fedora 20 host has SELinux enforcing, and the same issue occurs when set to permissive.  No AVCs are seen.

Version-Release number of selected component (if applicable):
docker-io-0.11.1-1.fc20.x86_64
kernel-3.14.2-200.fc20.x86_64

How reproducible:
Always

Steps to Reproduce:
1. docker pull centos
2. docker run -t centos /usr/sbin/useradd test

Actual results:
useradd: failure while writing changes to /etc/passwd

Expected results:
no output

Additional info:

On 0.10.0, an strace of useradd shows:

open("/etc/group", O_RDONLY|O_CLOEXEC)  = 11
fstat(11, {st_mode=S_IFREG|0644, st_size=379, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feb5efe5000
read(11, "root:x:0:\nbin:x:1:bin,daemon\ndae"..., 4096) = 379
close(11)                               = 0
munmap(0x7feb5efe5000, 4096)            = 0
fchown(10, 500, 12)                     = 0
fchmod(10, 0660)                        = 0
fsync(10)                               = 0
close(10)                               = 0
fstat(6, {st_mode=S_IFREG|0644, st_size=670, ...}) = 0
gettid()                                = 14
open("/proc/self/task/14/attr/fscreate", O_RDONLY) = 10
read(10, "", 4095)                      = 0
close(10)                               = 0
gettid()                                = 14
open("/proc/self/task/14/attr/fscreate", O_RDWR) = 10
write(10, "system_u:object_r:file_t:s0\0", 28) = 28
close(10)                               = 0
fstat(6, {st_mode=S_IFREG|0644, st_size=670, ...}) = 0
umask(077)                              = 022
open("/etc/passwd-", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 10
umask(022)                              = 077
lseek(6, 0, SEEK_SET)                   = 0
read(6, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 670
fstat(10, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feb5efe5000
read(6, "", 4096)                       = 0
write(10, "root:x:0:0:root:/root:/bin/bash\n"..., 670) = 670

While on 0.11.1, strace shows:

open("/etc/group", O_RDONLY|O_CLOEXEC)  = 10
fstat(10, {st_mode=S_IFREG|0644, st_size=379, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2905a38000
read(10, "root:x:0:\nbin:x:1:bin,daemon\ndae"..., 4096) = 379
close(10)                               = 0
munmap(0x7f2905a38000, 4096)            = 0
fchown(9, 500, 12)                      = 0
fchmod(9, 0660)                         = 0
fsync(9)                                = 0
close(9)                                = 0
fstat(5, {st_mode=S_IFREG|0644, st_size=675, ...}) = 0
gettid()                                = 30
open("/proc/self/task/30/attr/fscreate", O_RDONLY) = 9
read(9, "", 4095)                       = 0
close(9)                                = 0
gettid()                                = 30
open("/proc/self/task/30/attr/fscreate", O_RDWR) = -1 EROFS (Read-only file system)
write(2, "useradd: failure while writing c"..., 54useradd: failure while writing changes to /etc/passwd
) = 54

--- Additional comment from Dominic Cleal on 2014-05-09 07:13:07 EDT ---

useradd is just calling libselinux's setfscreatecon, which is being blocked.  On 0.11.1, this library call fails, on 0.10.0/0.9 it works.

--- Additional comment from Paul Morgan on 2014-05-09 09:55:15 EDT ---

when selinux=disabled, i cannot reproduce the bug.

when selinux is enabled (permissive), the bug is always reproducible.

i reproduced using two docker service configs:

* first, use default systemd unit
* second, add `--selinux-enabled` as described at
  http://blog.docker.io/2014/05/docker-0-11-release-candidate-for-1-0/

override default systemd unit...

# cp /usr/lib/systemd/system/docker.service /etc/systemd/system/

# vim /etc/systemd/system/docker.service
# cat !$ 
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
After=network.target

[Service]
ExecStart=/usr/bin/docker -d --selinux-enabled
Restart=on-failure
LimitNOFILE=1048576
LimitNPROC=1048576

[Install]
WantedBy=multi-user.target


# systemctl restart docker.service 
# systemctl status docker.service 
docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; enabled)
   Active: active (running) since Fri 2014-05-09 09:52:25 EDT; 4s ago
     Docs: http://docs.docker.io
 Main PID: 997 (docker)
   CGroup: /system.slice/docker.service
           └─997 /usr/bin/docker -d --selinux-enabled


other system info:

$ rpm -q selinux-policy-targeted
selinux-policy-targeted-3.12.1-158.fc20.noarch

$ uname -a
Linux f20-01.example.com 3.13.9-200.fc20.x86_64 #1 SMP Fri Apr 4 12:13:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ rpm -q docker-io
docker-io-0.11.1-1.fc20.x86_64

$ docker version
Client version: 0.11.1
Client API version: 1.11
Go version (client): go1.2.1
Git commit (client): fb99f99/0.11.1
Server version: 0.11.1
Server API version: 1.11
Git commit (server): fb99f99/0.11.1
Go version (server): go1.2.1
Last stable version: 0.11.1

$ docker images | grep '^centos'
centos                centos6             0b443ba03958        3 weeks ago         297.6 MB
centos                latest              0b443ba03958        3 weeks ago         297.6 MB
centos                6.4                 539c0211cd76        13 months ago       300.6 MB

--- Additional comment from Lokesh Mandvekar on 2014-05-09 13:41:51 EDT ---

I have a new scratch build http://kojipkgs.fedoraproject.org//work/tasks/2230/6832230/docker-io-0.11.1-3.fc20.x86_64.rpm 


with this build, this error doesn't occur with fedora:20, but still does with centos

dwalsh, comments?

--- Additional comment from Dominic Cleal on 2014-05-09 13:45:57 EDT ---

Comparing the straces between el6 and fedora:20, I don't see any of the same accesses to attr/fscreate on f20 that are in the bug description.  The source of shadow-utils between el6 & f20 looks very different too, I can't see any setfscreatecon calls in the useradd code path.

--- Additional comment from Daniel Walsh on 2014-05-09 16:51:06 EDT ---

The problem is inside the container it sees SELinux as being enabled, which is the bug.

If you do id -Z, does it complain inside the container?

 docker run --rm -t -i fedora sh
sh-4.2# id -Z
id: --context (-Z) works only on an SELinux-enabled kernel
sh-4.2# mount | grep /sys
sysfs on /sys type sysfs (ro,relatime,seclabel)


SELinux sees the container as being disabled since /sys/fs/selinux is mounted as read/only, this will tell useradd NOT to try to do any SELinux stuff while in the container.

--- Additional comment from Fedora Update System on 2014-05-10 00:04:23 EDT ---

docker-io-0.11.1-3.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/docker-io-0.11.1-3.fc20

--- Additional comment from Dominic Cleal on 2014-05-10 08:01:14 EDT ---

(In reply to Daniel Walsh from comment #5)
> The problem is inside the container it sees SELinux as being enabled, which
> is the bug.
> 
> If you do id -Z, does it complain inside the container?

No, it runs and reports a context.

>  docker run --rm -t -i fedora sh
> sh-4.2# id -Z
> id: --context (-Z) works only on an SELinux-enabled kernel
> sh-4.2# mount | grep /sys
> sysfs on /sys type sysfs (ro,relatime,seclabel)

$ rpm -q docker-io
docker-io-0.9.1-1.fc20.x86_64
$ docker run -i -t centos /bin/bash
bash-4.1# id -Z
system_u:system_r:docker_t:s0
bash-4.1# mount | grep sys
sysfs on /sys type sysfs (rw,seclabel,nosuid,nodev,noexec,relatime)

$ rpm -q docker-io
docker-io-0.11.1-3.fc20.x86_64
$ docker run -i -t centos /bin/bash
bash-4.1# id -Z
system_u:system_r:svirt_lxc_net_t:s0:c231,c400
bash-4.1# mount | grep /sys
sysfs on /sys type sysfs (ro,seclabel,relatime)

> SELinux sees the container as being disabled since /sys/fs/selinux is
> mounted as read/only, this will tell useradd NOT to try to do any SELinux
> stuff while in the container.

/sys is correctly read-only as you expected, but it seems useradd's still doing SELinux stuff then.  These packages are installed inside the EL6 container:

libselinux-2.0.94-5.3.el6_4.1.x86_64
libselinux-utils-2.0.94-5.3.el6_4.1.x86_64
shadow-utils-4.1.4.2-13.el6.x86_64

Calling is_selinux_enabled() on Fedora is returning 0, while on EL6 it's returning 1.  Another difference - on Fedora, getenforce returns "Disabled" but on EL6 it prints:

# getenforce
getenforce:  getenforce() failedbash-4.1# 

/selinux exists within the container, but nothing is actually mounted there.  It appears to be simply a directory on the root filesystem (/selinux/booleans exists as an empty dir).  No other SELinux mounts are visible.

Looking at libselinux-2.0.94, I think it's seeing selinuxfs listed in /proc/filesystems and assuming SELinux is enabled because of this.  libselinux-2.2.1 on F20 doesn't seem to have this code.

libselinux-2.0.94/src/enabled.c:
        /* Drop back to detecting it the long way. */
        fp = fopen("/proc/filesystems", "r");
        if (!fp)
                return -1;

        __fsetlocking(fp, FSETLOCKING_BYCALLER);
        while ((num = getline(&buf, &len, fp)) != -1) {
                if (strstr(buf, "selinuxfs")) {
                        enabled = 1;
                        break;
                }
        }

# grep selinux /proc/filesystems 
nodev	selinuxfs

(All the above was tested with docker-io-0.11.1-3.fc20)

--- Additional comment from Fedora Update System on 2014-05-12 01:28:00 EDT ---

Package docker-io-0.11.1-3.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing docker-io-0.11.1-3.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-6281/docker-io-0.11.1-3.fc20
then log in and leave karma (feedback).

--- Additional comment from Dominic Cleal on 2014-05-12 06:34:11 EDT ---

(In reply to Dominic Cleal from comment #7)
> Looking at libselinux-2.0.94, I think it's seeing selinuxfs listed in
> /proc/filesystems and assuming SELinux is enabled because of this. 
> libselinux-2.2.1 on F20 doesn't seem to have this code.

Bug #835146 (against EL6) seems to confirm this, suggesting a backport of the patch that removes the /proc/filesystems based check.

Comment 1 Daniel Walsh 2014-05-12 13:28:58 UTC
Libselinux needs to check if the /selinux or /sys/fs/selinux is mounted as read/only and then say SELinux is disabled.  This would stop SELinux aware applications from attempting to do SELinux actions while in a container.

Comment 9 errata-xmlrpc 2014-10-14 06:41:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1469.html

Comment 10 petrkuzel 2022-02-21 17:14:55 UTC
This bug is used as exemplary for The limits of compatibility and supportability with containers <https://www.redhat.com/en/blog/limits-compatibility-and-supportability-containers>.

I understand that for the same image and the same hosting kernel, the docker service upgrade impacted behaviour of the container.
Despite the docker service upgrade impacted containers it was not seen as a regression in the docker service but it was evaluated
as that the upgrade detected a latent bug in the container and preserving a bug-to-bug backward compatibility was not beneficial was it, please?


Note You need to log in before you can comment on or make changes to this bug.