Bug 1896883

We are unable to ship another version of Docker at this time.  RHEL7 is in final support stages where only security fixes will get released.  Starting in RHEL 8.0, Docker was replaced by Podman and we encourage transitioning to Podman from Docker. 

Jindrich, do you know if this is the latest version of Docker?  If it's not, an upgrade might help, but I'm not sure on that.

This is latest version of Docker ,updated yesterday by Redhat,
https://access.redhat.com/errata/RHBA-2020:5059

Affected version: 
docker-1.13.1-203.git0be3e21.el7_9.x86_64

Uneffected version :
docker-1.13.1-162.git64e9980.el7_8

This update was shipped yesterday by redhat and seems like this bug was introdused  while providing the swcurity update.

That's why requesting to fix the bug,
https://access.redhat.com/errata/RHBA-2020:5059

Kir,

I see this change between 64e9980 and 0be3e21:

diff -rub docker-64e9980da375aae15b467ec980bce898541fd356/vendor/github.com/opencontainers/runc/libcontainer/label/label_selinux.go docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/vendor/github.com/opencontainers/runc/libcontainer/label/label_selinux.go
--- docker-64e9980da375aae15b467ec980bce898541fd356/vendor/github.com/opencontainers/runc/libcontainer/label/label_selinux.go   2020-02-07 14:38:36.000000000 +0100
+++ docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/vendor/github.com/opencontainers/runc/libcontainer/label/label_selinux.go   2020-04-15 14:49:24.000000000 +0200
@@ -24,17 +24,22 @@
 // the container.  A list of options can be passed into this function to alter
 // the labels.  The labels returned will include a random MCS String, that is
 // guaranteed to be unique.
-func InitLabels(options []string) (string, string, error) {
+func InitLabels(options []string) (plabel string, mlabel string, Err error) {
        if !selinux.SelinuxEnabled() {
                return "", "", nil
        }
        processLabel, mountLabel := selinux.GetLxcContexts()
        if processLabel != "" {
+               defer func() {
+                       if Err != nil {
+                               UnreserveLabel(mountLabel)
+                       }
+               }()
                pcon := selinux.NewContext(processLabel)
                mcon := selinux.NewContext(mountLabel)
                for _, opt := range options {
                        if opt == "disable" {
-                               return "", "", nil
+                               return "", mountLabel, nil
                        }
                        if i := strings.Index(opt, ":"); i == -1 {
                                return "", "", fmt.Errorf("Bad label option %q, valid options 'disable' or \n'user, role, level, type' followed by ':' and a value", opt)


Could it cause a regression seen in this bug?

> I see this change between 64e9980 and 0be3e21:
> ...
> Could it cause a regression seen in this bug?

This change is from https://github.com/projectatomic/docker/pull/370 (and https://github.com/projectatomic/docker/pull/372), which is a backport of https://github.com/opencontainers/selinux/pull/35.

I can't see how this might result is whatever the customer is observing.

@pradpan2 you are saying you use `-v /usr/local/quay/storage:/datastorage:Z` argument to docker run, and you show `ls -ld /conf/stack` (presumably inside the container). How is /conf/stack path inside container related to /datastorage (the mount point)?

(In reply to Kir Kolyshkin from comment #7)
> This change is from https://github.com/projectatomic/docker/pull/370 (and
> https://github.com/projectatomic/docker/pull/372), which is a backport of
> https://github.com/opencontainers/selinux/pull/35.
> 
> I can't see how this might result is whatever the customer is observing.

I think it's because I also backported in the `mountLabel` listed here:

https://github.com/opencontainers/selinux/pull/35/commits/fd7b61c28f9e134d36a5911eb1550f2704e4bf8e#diff-412ed5386ecf7a52cd2bd637f5c432b0b2ba439c6e05ac6d0dbdb4ce90647336R42

Actually, Ulrich pointed this out to me when I applied the backport here and I simply didn't notice it (private bugzilla comment below):

https://bugzilla.redhat.com/show_bug.cgi?id=1734482#c21

This still exists in the go-selinux package too, here in InitLabels():

https://github.com/opencontainers/selinux/blob/master/go-selinux/label/label_linux.go#L50

The key is, when `--security-opt label=disable` is specified (or `--privileged`, which does the same security option) in Docker, it still carries the label across from InitMount and it is applied when :Z is specified, so anytime anyone applies :Z it isn't properly ignored.

This is very problematic for OpenShift 3.11 clusters still running Docker; they're not able to drop the :Z flag and the kubelet doesn't support removing :Z to stop the relabeling on large volumes until very, very much later:

https://github.com/kubernetes/enhancements/pull/1621

Proof this is the problem:

# rpm -q docker
docker-1.13.1-203.git0be3e21.el7_9.x86_64

Attach to the running Docker process:

# /root/go/bin/dlv attach $(pidof dockerd-current) /usr/bin/dockerd-current

Put a breakpoint on InitLabels:

(dlv) b InitLabels
Breakpoint 1 set at 0x8f7768 for github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/label.InitLabels() /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/label/label_selinux.go:27

Launch a container in another TTY with the proper options:

# mkdir -p /root/myvolume
# touch /root/myvolume/mycoolfile
# ls -lattrZ /root/myvolume/mycoolfile
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 /root/myvolume/mycoolfile

# docker run -d --name testyboi --security-opt label=disable -v /root/myvolume:/mnt:Z docker.io/nginx

(dlv) c
> github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/label.InitLabels() /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/label/label_selinux.go:27 (hits goroutine(478):1 total:1) (PC: 0x8f7768)
Warning: debugging optimized function
    22:
    23: // InitLabels returns the process label and file labels to be used within
    24: // the container.  A list of options can be passed into this function to alter
    25: // the labels.  The labels returned will include a random MCS String, that is
    26: // guaranteed to be unique.
=>  27: func InitLabels(options []string) (plabel string, mlabel string, Err error) {
    28:         if !selinux.SelinuxEnabled() {
    29:                 return "", "", nil
    30:         }
    31:         processLabel, mountLabel := selinux.GetLxcContexts()
    32:         if processLabel != "" {

(dlv) bt
 0  0x00000000008f7768 in github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/label.InitLabels
    at /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/label/label_selinux.go:27
 1  0x00000000011e65a9 in github.com/docker/docker/daemon.parseSecurityOpt
    at /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/daemon/daemon_unix.go:200
 2  0x00000000011cbf11 in github.com/docker/docker/daemon.(*Daemon).setSecurityOptions
    at /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/daemon/container.go:186
 3  0x00000000011d9091 in github.com/docker/docker/daemon.(*Daemon).create
    at /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/daemon/create.go:110
 4  0x00000000011d8a14 in github.com/docker/docker/daemon.(*Daemon).containerCreate
    at /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/daemon/create.go:61
 5  0x00000000011d87ad in github.com/docker/docker/daemon.(*Daemon).ContainerCreate
    at /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/daemon/create.go:34
- - - - 8< - - - -

Set a new breakpoint right before the return:

(dlv) b 40
Breakpoint 2 set at 0x8f78c4 for github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/label.InitLabels() /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/label/label_selinux.go:40

(dlv) c
> github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/label.InitLabels() /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/label/label_selinux.go:40 (hits goroutine(478):1 total:1) (PC: 0x8f78c4)
Warning: debugging optimized function
    35:                                 UnreserveLabel(mountLabel)
    36:                         }
    37:                 }()
    38:                 pcon := selinux.NewContext(processLabel)
    39:                 mcon := selinux.NewContext(mountLabel)
=>  40:                 for _, opt := range options {
    41:                         if opt == "disable" {
    42:                                 return "", mountLabel, nil
    43:                         }
    44:                         if i := strings.Index(opt, ":"); i == -1 {
    45:                                 return "", "", fmt.Errorf("Bad label option %q, valid options 'disable' or \n'user, role, level, type' followed by ':' and a value", opt)

(dlv) p options
[]string len: 1, cap: 1, ["disable"]

(dlv) p mountLabel
"system_u:object_r:svirt_sandbox_file_t:s0:c54,c837"

If we step a bit, we see we end up with the MountLabel not being nil, and later on down the road we'll use the container.MountLabel to set up the volume.

(dlv) s
> github.com/docker/docker/daemon.parseSecurityOpt() /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/daemon/daemon_unix.go:201 (PC: 0x11e660d)
Warning: debugging optimized function
   196:                         return fmt.Errorf("invalid --security-opt 2: %q", opt)
   197:                 }
   198:         }
   199:
   200:         container.ProcessLabel, container.MountLabel, err = label.InitLabels(labelOpts)
=> 201:         return err
   202: }
   203:
   204: func getBlkioThrottleDevices(devs []*blkiodev.ThrottleDevice) ([]specs.ThrottleDevice, error) {
   205:         var throttleDevices []specs.ThrottleDevice
   206:         var stat syscall.Stat_t

(dlv) p container.MountLabel
"system_u:object_r:svirt_sandbox_file_t:s0:c54,c837"

Here we use it:

(dlv) b setupMounts
Breakpoint 3 set at 0x122dffb for github.com/docker/docker/daemon.(*Daemon).setupMounts() /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/daemon/volumes_unix.go:28

(dlv) b github.com/docker/docker/volume.(*MountPoint).Setup
Breakpoint 4 set at 0x8f9df8 for github.com/docker/docker/volume.(*MountPoint).Setup() /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/volume/volume.go:129

(dlv) c
> github.com/docker/docker/volume.(*MountPoint).Setup() /usr/src/debug/docker-0be3e217c42ecf554bf5117bec9c832bd3f3b6fd/_build/src/github.com/docker/docker/volume/volume.go:129 (hits goroutine(488):1 total:1) (PC: 0x8f9df8)
Warning: debugging optimized function
   124:
   125: // Setup sets up a mount point by either mounting the volume if it is
   126: // configured, or creating the source directory if supplied.
   127: // The, optional, checkFun parameter allows doing additional checking
   128: // before creating the source directory on the host.
=> 129: func (m *MountPoint) Setup(mountLabel string, rootUID, rootGID int, checkFun func(m *MountPoint) error) (path string, err error) {
   130:         defer func() {
   131:                 if err == nil {
   132:                         if label.RelabelNeeded(m.Mode) {
   133:                                 sourcePath, err := filepath.EvalSymlinks(m.Source)
   134:                                 if err != nil {
   
(dlv) p m
*github.com/docker/docker/volume.MountPoint {
        Source: "/root/myvolume",
        Destination: "/mnt",
        RW: true,
        Name: "",
        Driver: "",
        Type: "bind",
        Volume: github.com/docker/docker/volume.Volume nil,
        Mode: "Z",
        Propagation: "rprivate",
        CopyData: false,
        ID: "",
        Spec: github.com/docker/docker/api/types/mount.Mount {
                Type: "bind",
                Source: "/root/myvolume",
                Target: "/mnt",
                ReadOnly: false,
                BindOptions: *github.com/docker/docker/api/types/mount.BindOptions nil,
                VolumeOptions: *github.com/docker/docker/api/types/mount.VolumeOptions nil,
                TmpfsOptions: *github.com/docker/docker/api/types/mount.TmpfsOptions nil,},}

(dlv) p mountLabel
"system_u:object_r:svirt_sandbox_file_t:s0:c54,c837"

If we returned "" as the second argument to InitLabels, as "disable" seems to want, this wouldn't happen.

@Kir thoughts?  I can open a PR against both projectatomic/docker and opencontainers/selinux to fix this if so.

Looks like Dan was already made aware of this?  Derrick showed me this commit.  This seems to fix it, we'll no longer pass the label when doing `disable`.  I haven't tested, just read it.

https://github.com/projectatomic/docker/pull/381/commits/090ced270e99f79b3428f976c1328f7cf3471793

(In reply to Robb Manes from comment #10)
> Looks like Dan was already made aware of this?  Derrick showed me this
> commit.  This seems to fix it, we'll no longer pass the label when doing
> `disable`.  I haven't tested, just read it.
> 
> https://github.com/projectatomic/docker/pull/381/commits/
> 090ced270e99f79b3428f976c1328f7cf3471793

FYI, this is the last remaining patch to follow fix Bug 1734482 as well

Yeah I think the best course of action is to merge https://github.com/projectatomic/docker/pull/381/ and release an update.

Give it a LGTM and I will merge.

hmm, I wonder if this issue could be the reason also for https://bugzilla.redhat.com/show_bug.cgi?id=1943323 (kubelet container makes docker crash when using with el7 docker -203 and -204 versions) ?

(In reply to Pasi Karkkainen from comment #14)
> hmm, I wonder if this issue could be the reason also for
> https://bugzilla.redhat.com/show_bug.cgi?id=1943323 (kubelet container makes
> docker crash when using with el7 docker -203 and -204 versions) ?

Pretty sure that's the case.

I can confirm this specific issue resolved with https://github.com/projectatomic/docker/pull/381/ after doing a build myself with -204 with a cherry-pick of https://github.com/projectatomic/docker/pull/381/.

# rpm -q docker
docker-1.13.1-204.git0be3e21bz1896883.el7.x86_64

# mkdir /root/newvolume

# touch /root/newvolume/myfile

# docker run -d --name testyboi --security-opt label=disable -v /root/newvolume:/mnt:Z docker.io/nginx

# ls -lZ /root/newvolume/
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 myfile

I can ALSO confirm the kubelet issues are resolved as well following the steps I put in https://bugzilla.redhat.com/show_bug.cgi?id=1943700.

$ rke up
- - - - 8< - - - -
INFO[0146] Starting container [kubelet] on host [localhost], try #1
INFO[0146] [worker] Successfully started [kubelet] container on host [localhost]
- - - - 8< - - - -

$ docker inspect kubelet --format '{{ .State.Running }}'
true

Since the kubelet's container in Docker was blocked during the SELinux labeling phase (which I verified with dlv and gdb) it seems to work as expected now.

*** Bug 1943700 has been marked as a duplicate of this bug. ***

*** Bug 1943323 has been marked as a duplicate of this bug. ***

This bug has been fixed on docker-1.13.1-205.git7d71120.el7_9.

[root@kvm-04-guest09 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)

[root@kvm-04-guest09 ~]# rpm -q docker kernel
docker-1.13.1-205.git7d71120.el7_9.x86_64
kernel-3.10.0-1160.21.1.el7.x86_64

[root@kvm-04-guest09 ~]# mkdir /root/newvolume
[root@kvm-04-guest09 ~]# touch /root/newvolume/myfile
[root@kvm-04-guest09 ~]# docker run -d --name testyboi --security-opt label=disable -v /root/newvolume:/mnt:Z docker.io/nginx
Unable to find image 'docker.io/nginx:latest' locally
Trying to pull repository docker.io/library/nginx ...
latest: Pulling from docker.io/library/nginx
75646c2fb410: Pull complete                                                                                                                                                   6128033c842f: Pull complete                                                                                                                                                   71a81b5270eb: Pull complete                                                                                                                                                   b5fc821c48a1: Pull complete                                                                                                                                                   da3f514a6428: Pull complete                                                                                                                                                   3be359fed358: Pull complete                                                                                                                                                   Digest: sha256:9c3cf52fad3ac56ca35052afe5e9ddad01e7bee7630d6ef2d022dc2392c9a1d0
Status: Downloaded newer image for docker.io/nginx:latest
6f3ce8bdfb24e340ebf809766c5e591d36834d8a95755427570ab7e42c97a102

[root@kvm-04-guest09 ~]# ls -lZ /root/newvolume/
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 myfile

Hello Team,

Can we have a confirmation on whether this bug is present in docker-1.13.1-204 version as well or not?

Regards,
Ayush Garg

(In reply to aygarg from comment #24)
> Can we have a confirmation on whether this bug is present in
> docker-1.13.1-204 version as well or not?

I opened this issue against docker-1.13.1-203.git0be3e21.el7_9 and Alex noted it is fixed in:

(In reply to Alex Jia from comment #23)
> This bug has been fixed on docker-1.13.1-205.git7d71120.el7_9.

There is no official release yet, but anything between docker-1.13.1-203.git0be3e21.el7_9 and now that is available from Red Hat is still affected, until an errata link is provided in this bugzilla - once that link is provided, it should be available for download through normal channels and should be a version higher than the one Alex noted above.

(In reply to Robb Manes from comment #25)
> (In reply to aygarg from comment #24)
> > Can we have a confirmation on whether this bug is present in
> > docker-1.13.1-204 version as well or not?
> 
> I opened this issue against docker-1.13.1-203.git0be3e21.el7_9 and Alex
> noted it is fixed in:
> 
> (In reply to Alex Jia from comment #23)
> > This bug has been fixed on docker-1.13.1-205.git7d71120.el7_9.
> 
> There is no official release yet, but anything between
> docker-1.13.1-203.git0be3e21.el7_9 and now that is available from Red Hat is
> still affected, until an errata link is provided in this bugzilla - once
> that link is provided, it should be available for download through normal
> channels and should be a version higher than the one Alex noted above.

Thanks a lot Robb for replying.

I think then it's better to inform the customer to wait in my case until the fix is released instead of upgrading the package.

Ayush, yes, this bug it present in -204 version as well. The -205 was cut just to address this issue.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (docker bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1406