Bug 1294459 - Bind-mounting of volumes does not seems to be successful always [NEEDINFO]
Bind-mounting of volumes does not seems to be successful always
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: rhel-server-docker (Show other bugs)
7.2
Unspecified Unspecified
high Severity high
: rc
: ---
Assigned To: Frantisek Kluknavsky
atomic-bugs@redhat.com
: Extras
Depends On:
Blocks: 1293667 1293669 1294733
  Show dependency treegraph
 
Reported: 2015-12-28 05:06 EST by Humble Chirammal
Modified: 2016-10-05 12:34 EDT (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1293669
Environment:
Last Closed: 2016-09-27 08:24:17 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
fkluknav: needinfo? (hchiramm)


Attachments (Terms of Use)

  None (edit)
Comment 3 Anoop 2015-12-28 23:51:25 EST
This is impacting QE test progress Hitting this issues often.
Comment 4 Humble Chirammal 2016-01-02 00:25:43 EST
Some more isolation has been done here:

Everything **works as expected** when using a container image (http://docker-registry.usersys.redhat.com/#q=gluster/rhgs-3.1.0-3) which is built on top of rhel 7.1 base image.

--snip--

[root@dhcp37-135 /]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.1 (Maipo)
[root@dhcp37-135 /]# 
[root@dhcp37-135 /]# rpm -qa |grep systemd   
systemd-container-libs-208.20-6.el7.x86_64
systemd-container-208.20-6.el7.x86_64

--/snip--

Things go wrong when using RHEL 7.2 based container image , for ex: rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7 .

More details can be found @https://bugzilla.redhat.com/show_bug.cgi?id=1293669#c8
Comment 5 Humble Chirammal 2016-01-03 02:27:04 EST
Here is the excerpt from the setup:

Below mount points are available in atomic host system:

/dev/mapper/RHGS_VG1-vol1             10G   33M   10G   1% /var/mnt/brick1
/dev/mapper/RHGS_VG2-vol2             10G   33M   10G   1% /var/mnt/brick2


-bash-4.2# docker -D run -d --privileged=true --net=host --name newnode1 -v /etc/glusterfs:/etc/glusterfs  -v /var/lib/glusterd:/var/lib/glusterd -v /var/mnt/brick1:/b1 -v /var/mnt/brick2:/b2 -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /dev:/dev rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7
3cee47bc7f7bf57b2d6c3b3e744aeb13d431d1428937cdabd3cb8b5bb25f83d7
DEBU[0001] End of CmdRun(), Waiting for hijack to finish. 
-bash-4.2# 


The container is started, however the volumes are not mounted inside the container.


-bash-4.2# docker ps
CONTAINER ID        IMAGE                                                                    COMMAND             CREATED             STATUS              PORTS               NAMES
3cee47bc7f7b        rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7   "/usr/sbin/init"    2 minutes ago       Up 2 minutes                            newnode1
-bash-4.2# docker inspect 3cee47bc7f7b

    "HostConfig": {
        "Binds": [
            "/etc/glusterfs:/etc/glusterfs",
            "/var/lib/glusterd:/var/lib/glusterd",
            "/var/mnt/brick1:/b1",
            "/var/mnt/brick2:/b2",
            "/sys/fs/cgroup:/sys/fs/cgroup:ro",
            "/dev:/dev"
        ],



   "Mounts": [
        {
            "Source": "/etc/glusterfs",
            "Destination": "/etc/glusterfs",
            "Mode": "",
            "RW": true
        },
        {
            "Source": "/var/lib/glusterd",
            "Destination": "/var/lib/glusterd",
            "Mode": "",
            "RW": true
        },
        {
            "Source": "/var/mnt/brick1",
            "Destination": "/b1",
            "Mode": "",
            "RW": true
        },
        {
            "Source": "/var/mnt/brick2",
            "Destination": "/b2",
            "Mode": "",
            "RW": true
        },
        {
            "Source": "/sys/fs/cgroup",
            "Destination": "/sys/fs/cgroup",
            "Mode": "ro",
            "RW": false
        },
        {
            "Source": "/dev",
            "Destination": "/dev",
            "Mode": "",
            "RW": true
        }
    ],

'mount' command output from the container.


-bash-4.2# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=3987572k,nr_inodes=996893,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/rhelah_dhcp37--135-root on /sysroot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/rhelah_dhcp37--135-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/rhelah_dhcp37--135-root on /var type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/rhelah_dhcp37--135-root on /usr type xfs (ro,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/sda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801324k,mode=700)


-bash-4.2# date
Sun Jan  3 03:27:47 IST 2016
-bash-4.2# 

LOGS:


Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Unmounting /var...
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com umount[38401]: umount: /sysroot: target is busy.
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com umount[38401]: (In some cases useful info about processes that use
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com umount[38401]: the device is found by lsof(8) or fuser(1))
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: sysroot.mount mount process exited, code=exited status=32
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Unit sysroot.mount is bound to inactive unit dev-mapper-rhelah_dhcp37\x2d\x2d135\x2droot.device. Stopping, too.
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: usr.mount mount process exited, code=exited status=32
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Unit usr.mount is bound to inactive unit dev-mapper-rhelah_dhcp37\x2d\x2d135\x2droot.device.



Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /usr.
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /sysroot.
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com umount[39897]: umount: /sysroot: target is busy.
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com umount[39897]: (In some cases useful info about processes that use
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com umount[39897]: the device is found by lsof(8) or fuser(1))
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /var.
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /usr.
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /sysroot.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /sysroot.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /var.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39909]: umount: /var: target is busy.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39909]: (In some cases useful info about processes that use
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39909]: the device is found by lsof(8) or fuser(1))
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /usr.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /var.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /sysroot.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39908]: umount: /usr: target is busy.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39908]: (In some cases useful info about processes that use
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39908]: the device is found by lsof(8) or fuser(1))
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39911]: umount: /var: target is busy.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39911]: (In some cases useful info about processes that use
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39911]: the device is found by lsof(8) or fuser(1))
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39912]: umount: /sysroot: target is busy.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39912]: (In some cases useful info about processes that use
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39912]: the device is found by lsof(8) or fuser(1))
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39913]: umount: /var: target is busy.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /var.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /usr.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /sysroot.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /var.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39914]: umount: /usr: target is busy.




-bash-4.2# journalctl -xn
-- Logs begin at Wed 2015-12-30 08:37:10 IST, end at Sun 2016-01-03 03:29:14 IST. --
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel:  [<ffffffff810b8c22>] ? default_wake_function+0x12/0x20
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel:  [<ffffffff8163cd68>] tty_ldisc_lock_pair_timeout+0x88/0x120
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel:  [<ffffffff813b644c>] tty_ldisc_hangup+0xcc/0x230
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel:  [<ffffffff813adbf4>] __tty_hangup+0x344/0x490
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel:  [<ffffffff813b04b5>] tty_ioctl+0x885/0xbc0
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel:  [<ffffffff811efe3b>] ? do_filp_open+0x4b/0xb0
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel:  [<ffffffff811f1ef5>] do_vfs_ioctl+0x2e5/0x4c0
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel:  [<ffffffff8128bc6e>] ? file_has_perm+0xae/0xc0
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel:  [<ffffffff811f2171>] SyS_ioctl+0xa1/0xc0
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel:  [<ffffffff81645909>] system_call_fastpath+0x16/0x1b


[root@dhcp37-135 /]# rpm -qa |grep systemd
systemd-sysv-219-19.el7.x86_64
systemd-libs-219-19.el7.x86_64
systemd-219-19.el7.x86_64
[root@dhcp37-135 /]# 


[root@dhcp37-135 /]# uname -r
3.10.0-327.el7.x86_64

[root@dhcp37-135 /]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@dhcp37-135 /]# 

"Device view" from the container:

[root@dhcp37-135 /]# lvs
  LV          VG                Attr       LSize  Pool       Origin Data%  Meta%  Move Log Cpy%Sync Convert
  thin_pool0  RHGS_VG1          twi-aotz-- 19.00g                   0.06   0.57                            
  vol1        RHGS_VG1          Vwi-aotz-- 10.00g thin_pool0        0.11                                   
  thin_pool0  RHGS_VG2          twi-aotz-- 19.00g                   0.06   0.57                            
  vol2        RHGS_VG2          Vwi-aotz-- 10.00g thin_pool0        0.11                                   
  docker-pool rhelah_dhcp37-135 twi-aot---  7.67g                   26.15  2.43                            
  root        rhelah_dhcp37-135 -wi-ao----  3.00g                                                          
  swap        rhelah_dhcp37-135 -wi-ao----  2.50g                                                          
[root@dhcp37-135 /]# vgs
  VG                #PV #LV #SN Attr   VSize  VFree  
  RHGS_VG1            1   2   0 wz--n- 20.00g 980.00m
  RHGS_VG2            1   2   0 wz--n- 20.00g 980.00m
  rhelah_dhcp37-135   1   3   0 wz--n- 24.70g  11.48g
[root@dhcp37-135 /]# pvs
  PV         VG                Fmt  Attr PSize  PFree  
  /dev/sda2  rhelah_dhcp37-135 lvm2 a--  24.70g  11.48g
  /dev/vda   RHGS_VG1          lvm2 a--  20.00g 980.00m
  /dev/vdb   RHGS_VG2          lvm2 a--  20.00g 980.00m
[root@dhcp37-135 /]# 


Once the container is started, the subjected mount points are unmounted/vanished from the atomic host system.  

-bash-4.2# ps aux |grep docker
root       1064  0.0  0.2 1054592 22748 ?       Ssl   2015   1:38 /usr/bin/docker daemon --selinux-enabled --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/rhelah_dhcp37--135-docker--pool --add-registry docker-registry.usersys.redhat.com --insecure-registry docker-registry.usersys.redhat.com

-bash-4.2# rpm -qa |grep docker
docker-1.8.2-8.el7.x86_64
docker-selinux-1.8.2-8.el7.x86_64
python-docker-py-1.4.0-118.el7.noarch
-bash-4.2# 

-bash-4.2# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28
-bash-4.2# 

Here onwards, the docker commands will not work. 
For ex: docker stop <this container id> .
Comment 7 Humble Chirammal 2016-01-11 08:10:34 EST
Looks like we are hitting this issue https://bugzilla.redhat.com/show_bug.cgi?id=1285863 .
Comment 8 Cesar Sanchez 2016-01-13 15:12:41 EST
I'm having a similar issue with an el7 docker image.

I'm running some automated tests that rely on mounting some directories using the bind option.

It worked for months until now. I guess the most recent systemd updates broke my entire tests.

So, right now if you pull an el7 image (let's try the official centos):

1.  docker create --privileged=true -v /sys/fs/cgroup -v /run --entrypoint /usr/sbin/init docker.io/centos:centos7
2.  docker start <containerid>
3.  docker exec -it <containerid> mkdir /tmp/test
4.  docker exec -it <containerid> mount -o bind /etc /tmp/test
The output of the next 2 command should be the same given that they should be the same source dir:
5   docker exec -it <containerid> find /etc -type f | wc -l
174
6.  docker exec -it <containerid> find /tmp/test -type f | wc -l
0

So, it gets frustrating at this point, but finding this BZ gave me a little bit hope :)

I setup a little test system with this:

[root@tsbuilder3 baseimage-el6]# docker info
Containers: 3
Images: 37
Server Version: 1.9.1-fc23
Storage Driver: devicemapper
 Pool Name: docker-253:0-567770-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 21.47 GB
 Backing Filesystem: 
 Data file: /dev/docker/containers
 Metadata file: /dev/docker/metadata
 Data Space Used: 2.893 GB
 Data Space Total: 192.2 GB
 Data Space Available: 189.3 GB
 Metadata Space Used: 4.415 MB
 Metadata Space Total: 17.05 GB
 Metadata Space Available: 17.04 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.109 (2015-09-22)
Execution Driver: native-0.2
Logging Driver: journald
Kernel Version: 4.2.7-300.fc23.x86_64
Operating System: Fedora 23 (Twenty Three)
CPUs: 4
Total Memory: 3.86 GiB
Name: XXXXXXXXXXXX
ID: XXXXXXXXXX

docker inspect <CONTAINERID>
[
{
    "Id": "<CONTAINERID>",
    "Created": "2016-01-13T19:34:32.062398984Z",
    "Path": "/usr/sbin/init",
    "Args": [],
    "State": {
        "Status": "running",
        "Running": true,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 1918,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2016-01-13T19:34:39.579629283Z",
        "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "7a21760c41c742ab3cfaacb4b0fb7a03406781cacd8024ad28526c5ba97839cd",
    "ResolvConfPath": "/var/lib/docker/containers/fd9b62d7154471495f3fd6816e7c36399a86a9fb44ada91fa90b90228974940b/resolv.conf",
    "HostnamePath": "/var/lib/docker/containers/fd9b62d7154471495f3fd6816e7c36399a86a9fb44ada91fa90b90228974940b/hostname",
    "HostsPath": "/var/lib/docker/containers/fd9b62d7154471495f3fd6816e7c36399a86a9fb44ada91fa90b90228974940b/hosts",
    "LogPath": "",
    "Name": "/pensive_mcclintock",
    "RestartCount": 0,
    "Driver": "devicemapper",
    "ExecDriver": "native-0.2",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "ExecIDs": [
        "31d846f24e228729b2d5c3f9a38e324ab6cc23651b666f0dd24b9666161ac54d"
    ],
    "HostConfig": {
        "Binds": null,
        "ContainerIDFile": "",
        "LxcConf": [],
        "Memory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "KernelMemory": 0,
        "CpuShares": 0,
        "CpuPeriod": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "CpuQuota": 0,
        "BlkioWeight": 0,
        "OomKillDisable": false,
        "MemorySwappiness": -1,
        "Privileged": true,
        "PortBindings": {},
        "Links": null,
        "PublishAllPorts": false,
        "Dns": [],
        "DnsOptions": [],
        "DnsSearch": [],
        "ExtraHosts": null,
        "VolumesFrom": null,
        "Devices": [],
        "NetworkMode": "default",
        "IpcMode": "",
        "PidMode": "",
        "UTSMode": "",
        "CapAdd": null,
        "CapDrop": null,
        "GroupAdd": null,
        "RestartPolicy": {
            "Name": "no",
            "MaximumRetryCount": 0
        },
        "SecurityOpt": null,
        "ReadonlyRootfs": false,
        "Ulimits": null,
        "LogConfig": {
            "Type": "journald",
            "Config": {}
        },
        "CgroupParent": "",
        "ConsoleSize": [
            0,
            0
        ],
        "VolumeDriver": "",
        "ShmSize": 67108864
    },
    "GraphDriver": {
        "Name": "devicemapper",
        "Data": {
            "DeviceId": "290",
            "DeviceName": "docker-253:0-567770-fd9b62d7154471495f3fd6816e7c36399a86a9fb44ada91fa90b90228974940b",
            "DeviceSize": "21474836480"
        }
    },
    "Mounts": [
        {
            "Name": "717d5c9b85ef494b7bd0520f2d1a8425a96d99c08d00b13f742161b604f4ce73",
            "Source": "/var/lib/docker/volumes/717d5c9b85ef494b7bd0520f2d1a8425a96d99c08d00b13f742161b604f4ce73/_data",
            "Destination": "/sys/fs/cgroup",
            "Driver": "local",
            "Mode": "",
            "RW": true
        },
        {
            "Name": "43491f02e24e460d41b91b1becca20655c173a47d6f5d86ad1c1ecaee0af8c56",
            "Source": "/var/lib/docker/volumes/43491f02e24e460d41b91b1becca20655c173a47d6f5d86ad1c1ecaee0af8c56/_data",
            "Destination": "/run",
            "Driver": "local",
            "Mode": "",
            "RW": true
        }
    ],
    "Config": {
        "Hostname": "fd9b62d71544",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": true,
        "AttachStderr": true,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": null,
        "Cmd": null,
        "Image": "docker.io/centos:centos7",
        "Volumes": {
            "/run": {},
            "/sys/fs/cgroup": {}
        },
        "WorkingDir": "",
        "Entrypoint": [
            "/usr/sbin/init"
        ],
        "OnBuild": null,
        "Labels": {
            "license": "GPLv2",
            "name": "CentOS Base Image",
            "vendor": "CentOS"
        },
        "StopSignal": "SIGTERM"
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "95d6b4432e7eccd68e78d682304c4edb0824c54c4399b681fdc1efe57a8317e6",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {},
        "SandboxKey": "/var/run/docker/netns/95d6b4432e7e",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "363bdca9844a6c0b0f153e3ac4ed246c8eee368d082ddef597fbd947dc890998",
        "Gateway": "172.17.0.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "172.17.0.2",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "MacAddress": "02:42:ac:11:00:02",
        "Networks": {
            "bridge": {
                "EndpointID": "363bdca9844a6c0b0f153e3ac4ed246c8eee368d082ddef597fbd947dc890998",
                "Gateway": "172.17.0.1",
                "IPAddress": "172.17.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:42:ac:11:00:02"
            }
        }
    }
}
]

Systemd rpm version:

systemd-219-19.el7.x86_64

Thanks,

-Cesar
Comment 9 hchen 2016-01-14 09:21:42 EST
Looks this is a systemd issue. I am able to run your find test successfully when I change entrypoint to bash rather than init.
Comment 10 hchen 2016-01-14 09:32:16 EST
When systemd is running inside the container and a bind mount is issued, I find the following message surfaces the problem:

# journalctl |grep systemd |grep mount
Jan 14 14:25:46 5d55a45afec8 systemd[1]: Unit etc-hostname.mount is bound to inactive unit dev-md2.device. Stopping, too.
Jan 14 14:25:46 5d55a45afec8 systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
Jan 14 14:25:46 5d55a45afec8 systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point.
Jan 14 14:25:46 5d55a45afec8 systemd[1]: Starting Remount Root and Kernel File Systems...
Jan 14 14:25:46 5d55a45afec8 systemd[1]: Unit etc-hostname.mount is bound to inactive unit dev-md2.device. Stopping, too.
Jan 14 14:25:46 5d55a45afec8 systemd[1]: Unit etc-hosts.mount is bound to inactive unit dev-md2.device. Stopping, too.
Jan 14 14:25:46 5d55a45afec8 systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
Jan 14 14:25:46 5d55a45afec8 systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point.
Jan 14 14:25:46 5d55a45afec8 systemd[1]: Starting Remount Root and Kernel File Systems...
Jan 14 14:25:46 5d55a45afec8 systemd-remount-fs[24]: mount: can't find UUID=1c74ece3-3844-44a6-a1b6-f8defa8f03fd
Jan 14 14:25:46 5d55a45afec8 systemd-remount-fs[24]: /bin/mount for / exited with exit status 1.
Jan 14 14:25:46 5d55a45afec8 systemd[1]: systemd-remount-fs.service: main process exited, code=exited, status=1/FAILURE
Jan 14 14:25:46 5d55a45afec8 systemd[1]: Failed to start Remount Root and Kernel File Systems.
Jan 14 14:25:46 5d55a45afec8 systemd[1]: Unit systemd-remount-fs.service entered failed state.
Jan 14 14:25:46 5d55a45afec8 systemd[1]: systemd-remount-fs.service failed.
Jan 14 14:26:18 5d55a45afec8 systemd[1]: Unit tmp-test.mount is bound to inactive unit dev-mapper-docker\x2d9:2\x2d9568392\x2d5d55a45afec8656c43dc84753ac62e00f7860e5a88b30c344713469a968e6b1d.device. Stopping, too.
Jan 14 14:26:18 5d55a45afec8 systemd[1]: Unmounting /tmp/test...
Jan 14 14:26:18 5d55a45afec8 systemd[1]: Unmounted /tmp/test.
Comment 11 Humble Chirammal 2016-01-14 11:08:43 EST
@All, we tried the fix mentioned @ https://bugzilla.redhat.com/show_bug.cgi?id=1294459#c7  and it seems that it resolves our issues.
Comment 12 Humble Chirammal 2016-01-21 09:32:39 EST
(In reply to Humble Chirammal from comment #11)
> @All, we tried the fix mentioned @
> https://bugzilla.redhat.com/show_bug.cgi?id=1294459#c7  and it seems that it
> resolves our issues.

Unfortunately we still hit some part of the issue mentioned in this bugzilla. That said, when spawning containers from  image which has systemd fixes mentioned in c#11,  the bind mounted volumes are getting unmounted from the atomic hosts. however it noticed that, the volumes are mounted inside the container. I will update the bugzilla with the details soon.
Comment 13 Mohamed Ashiq 2016-01-22 06:04:40 EST
(In reply to Humble Chirammal from comment #12)
> (In reply to Humble Chirammal from comment #11)
> > @All, we tried the fix mentioned @
> > https://bugzilla.redhat.com/show_bug.cgi?id=1294459#c7  and it seems that it
> > resolves our issues.
> 
> Unfortunately we still hit some part of the issue mentioned in this
> bugzilla. That said, when spawning containers from  image which has systemd
> fixes mentioned in c#11,  the bind mounted volumes are getting unmounted
> from the atomic hosts. however it noticed that, the volumes are mounted
> inside the container. I will update the bugzilla with the details soon.

When we are bind mounting volume into the rhgs container from atomic host, The issues we face are
1) LV unmount happening in the atomic host automatically
https://bugzilla.redhat.com/show_bug.cgi?id=1294776

2) docker stop/kill hung 
https://bugzilla.redhat.com/show_bug.cgi?id=1293667

we tried remounting the LV's before doing docker stop, Still docker stop hangs.
we had to restart docker service to run docker commands.

The complete setup and issue is been recorded here:
https://bugzilla.redhat.com/show_bug.cgi?id=1293667#c14

-bash-4.2# rpm -qa | grep docker
docker-1.8.2-10.el7.x86_64
docker-selinux-1.8.2-10.el7.x86_64
python-docker-py-1.4.0-118.el7.noarch

-bash-4.2# rpm -qa | grep systemd
systemd-sysv-219-19.el7.x86_64
systemd-219-19.el7.x86_64
systemd-libs-219-19.el7.x86_64
Comment 14 Daniel Walsh 2016-08-20 05:15:38 EDT
Could we get an update on this bug, Are you still seeing these issues?
Comment 15 Daniel Walsh 2016-09-27 08:24:17 EDT
I am going to assume this is fixed in the latest versions, since we never heard about it again.

Note You need to log in before you can comment on or make changes to this bug.