Bug 1294459
| Summary: | Bind-mounting of volumes does not seems to be successful always | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Humble Chirammal <hchiramm> |
| Component: | rhel-server-container | Assignee: | Frantisek Kluknavsky <fkluknav> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | atomic-bugs <atomic-bugs> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.2 | CC: | annair, asrivast, csanchez, dwalsh, fkluknav, hchen, hchiramm, lmiksik, mliyazud, pprakash, rcyriac, sankarshan, ssampat |
| Target Milestone: | rc | Keywords: | Extras |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1293669 | Environment: | |
| Last Closed: | 2016-09-27 12:24:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1293667, 1293669, 1294733 | ||
|
Comment 3
Anoop
2015-12-29 04:51:25 UTC
Some more isolation has been done here: Everything **works as expected** when using a container image (http://docker-registry.usersys.redhat.com/#q=gluster/rhgs-3.1.0-3) which is built on top of rhel 7.1 base image. --snip-- [root@dhcp37-135 /]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.1 (Maipo) [root@dhcp37-135 /]# [root@dhcp37-135 /]# rpm -qa |grep systemd systemd-container-libs-208.20-6.el7.x86_64 systemd-container-208.20-6.el7.x86_64 --/snip-- Things go wrong when using RHEL 7.2 based container image , for ex: rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7 . More details can be found @https://bugzilla.redhat.com/show_bug.cgi?id=1293669#c8 Here is the excerpt from the setup:
Below mount points are available in atomic host system:
/dev/mapper/RHGS_VG1-vol1 10G 33M 10G 1% /var/mnt/brick1
/dev/mapper/RHGS_VG2-vol2 10G 33M 10G 1% /var/mnt/brick2
-bash-4.2# docker -D run -d --privileged=true --net=host --name newnode1 -v /etc/glusterfs:/etc/glusterfs -v /var/lib/glusterd:/var/lib/glusterd -v /var/mnt/brick1:/b1 -v /var/mnt/brick2:/b2 -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /dev:/dev rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7
3cee47bc7f7bf57b2d6c3b3e744aeb13d431d1428937cdabd3cb8b5bb25f83d7
DEBU[0001] End of CmdRun(), Waiting for hijack to finish.
-bash-4.2#
The container is started, however the volumes are not mounted inside the container.
-bash-4.2# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3cee47bc7f7b rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7 "/usr/sbin/init" 2 minutes ago Up 2 minutes newnode1
-bash-4.2# docker inspect 3cee47bc7f7b
"HostConfig": {
"Binds": [
"/etc/glusterfs:/etc/glusterfs",
"/var/lib/glusterd:/var/lib/glusterd",
"/var/mnt/brick1:/b1",
"/var/mnt/brick2:/b2",
"/sys/fs/cgroup:/sys/fs/cgroup:ro",
"/dev:/dev"
],
"Mounts": [
{
"Source": "/etc/glusterfs",
"Destination": "/etc/glusterfs",
"Mode": "",
"RW": true
},
{
"Source": "/var/lib/glusterd",
"Destination": "/var/lib/glusterd",
"Mode": "",
"RW": true
},
{
"Source": "/var/mnt/brick1",
"Destination": "/b1",
"Mode": "",
"RW": true
},
{
"Source": "/var/mnt/brick2",
"Destination": "/b2",
"Mode": "",
"RW": true
},
{
"Source": "/sys/fs/cgroup",
"Destination": "/sys/fs/cgroup",
"Mode": "ro",
"RW": false
},
{
"Source": "/dev",
"Destination": "/dev",
"Mode": "",
"RW": true
}
],
'mount' command output from the container.
-bash-4.2# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=3987572k,nr_inodes=996893,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/rhelah_dhcp37--135-root on /sysroot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/rhelah_dhcp37--135-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/rhelah_dhcp37--135-root on /var type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/rhelah_dhcp37--135-root on /usr type xfs (ro,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/sda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801324k,mode=700)
-bash-4.2# date
Sun Jan 3 03:27:47 IST 2016
-bash-4.2#
LOGS:
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Unmounting /var...
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com umount[38401]: umount: /sysroot: target is busy.
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com umount[38401]: (In some cases useful info about processes that use
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com umount[38401]: the device is found by lsof(8) or fuser(1))
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: sysroot.mount mount process exited, code=exited status=32
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Unit sysroot.mount is bound to inactive unit dev-mapper-rhelah_dhcp37\x2d\x2d135\x2droot.device. Stopping, too.
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: usr.mount mount process exited, code=exited status=32
Jan 03 03:26:22 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Unit usr.mount is bound to inactive unit dev-mapper-rhelah_dhcp37\x2d\x2d135\x2droot.device.
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /usr.
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /sysroot.
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com umount[39897]: umount: /sysroot: target is busy.
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com umount[39897]: (In some cases useful info about processes that use
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com umount[39897]: the device is found by lsof(8) or fuser(1))
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /var.
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /usr.
Jan 03 03:26:23 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /sysroot.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /sysroot.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /var.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39909]: umount: /var: target is busy.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39909]: (In some cases useful info about processes that use
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39909]: the device is found by lsof(8) or fuser(1))
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /usr.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /var.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /sysroot.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39908]: umount: /usr: target is busy.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39908]: (In some cases useful info about processes that use
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39908]: the device is found by lsof(8) or fuser(1))
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39911]: umount: /var: target is busy.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39911]: (In some cases useful info about processes that use
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39911]: the device is found by lsof(8) or fuser(1))
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39912]: umount: /sysroot: target is busy.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39912]: (In some cases useful info about processes that use
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39912]: the device is found by lsof(8) or fuser(1))
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39913]: umount: /var: target is busy.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /var.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /usr.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /sysroot.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com systemd[1]: Failed unmounting /var.
Jan 03 03:26:24 dhcp37-135.lab.eng.blr.redhat.com umount[39914]: umount: /usr: target is busy.
-bash-4.2# journalctl -xn
-- Logs begin at Wed 2015-12-30 08:37:10 IST, end at Sun 2016-01-03 03:29:14 IST. --
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel: [<ffffffff810b8c22>] ? default_wake_function+0x12/0x20
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel: [<ffffffff8163cd68>] tty_ldisc_lock_pair_timeout+0x88/0x120
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel: [<ffffffff813b644c>] tty_ldisc_hangup+0xcc/0x230
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel: [<ffffffff813adbf4>] __tty_hangup+0x344/0x490
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel: [<ffffffff813b04b5>] tty_ioctl+0x885/0xbc0
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel: [<ffffffff811efe3b>] ? do_filp_open+0x4b/0xb0
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel: [<ffffffff811f1ef5>] do_vfs_ioctl+0x2e5/0x4c0
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel: [<ffffffff8128bc6e>] ? file_has_perm+0xae/0xc0
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel: [<ffffffff811f2171>] SyS_ioctl+0xa1/0xc0
Jan 03 03:29:14 dhcp37-135.lab.eng.blr.redhat.com kernel: [<ffffffff81645909>] system_call_fastpath+0x16/0x1b
[root@dhcp37-135 /]# rpm -qa |grep systemd
systemd-sysv-219-19.el7.x86_64
systemd-libs-219-19.el7.x86_64
systemd-219-19.el7.x86_64
[root@dhcp37-135 /]#
[root@dhcp37-135 /]# uname -r
3.10.0-327.el7.x86_64
[root@dhcp37-135 /]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@dhcp37-135 /]#
"Device view" from the container:
[root@dhcp37-135 /]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
thin_pool0 RHGS_VG1 twi-aotz-- 19.00g 0.06 0.57
vol1 RHGS_VG1 Vwi-aotz-- 10.00g thin_pool0 0.11
thin_pool0 RHGS_VG2 twi-aotz-- 19.00g 0.06 0.57
vol2 RHGS_VG2 Vwi-aotz-- 10.00g thin_pool0 0.11
docker-pool rhelah_dhcp37-135 twi-aot--- 7.67g 26.15 2.43
root rhelah_dhcp37-135 -wi-ao---- 3.00g
swap rhelah_dhcp37-135 -wi-ao---- 2.50g
[root@dhcp37-135 /]# vgs
VG #PV #LV #SN Attr VSize VFree
RHGS_VG1 1 2 0 wz--n- 20.00g 980.00m
RHGS_VG2 1 2 0 wz--n- 20.00g 980.00m
rhelah_dhcp37-135 1 3 0 wz--n- 24.70g 11.48g
[root@dhcp37-135 /]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 rhelah_dhcp37-135 lvm2 a-- 24.70g 11.48g
/dev/vda RHGS_VG1 lvm2 a-- 20.00g 980.00m
/dev/vdb RHGS_VG2 lvm2 a-- 20.00g 980.00m
[root@dhcp37-135 /]#
Once the container is started, the subjected mount points are unmounted/vanished from the atomic host system.
-bash-4.2# ps aux |grep docker
root 1064 0.0 0.2 1054592 22748 ? Ssl 2015 1:38 /usr/bin/docker daemon --selinux-enabled --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/rhelah_dhcp37--135-docker--pool --add-registry docker-registry.usersys.redhat.com --insecure-registry docker-registry.usersys.redhat.com
-bash-4.2# rpm -qa |grep docker
docker-1.8.2-8.el7.x86_64
docker-selinux-1.8.2-8.el7.x86_64
python-docker-py-1.4.0-118.el7.noarch
-bash-4.2#
-bash-4.2# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28
-bash-4.2#
Here onwards, the docker commands will not work.
For ex: docker stop <this container id> .
Looks like we are hitting this issue https://bugzilla.redhat.com/show_bug.cgi?id=1285863 . I'm having a similar issue with an el7 docker image.
I'm running some automated tests that rely on mounting some directories using the bind option.
It worked for months until now. I guess the most recent systemd updates broke my entire tests.
So, right now if you pull an el7 image (let's try the official centos):
1. docker create --privileged=true -v /sys/fs/cgroup -v /run --entrypoint /usr/sbin/init docker.io/centos:centos7
2. docker start <containerid>
3. docker exec -it <containerid> mkdir /tmp/test
4. docker exec -it <containerid> mount -o bind /etc /tmp/test
The output of the next 2 command should be the same given that they should be the same source dir:
5 docker exec -it <containerid> find /etc -type f | wc -l
174
6. docker exec -it <containerid> find /tmp/test -type f | wc -l
0
So, it gets frustrating at this point, but finding this BZ gave me a little bit hope :)
I setup a little test system with this:
[root@tsbuilder3 baseimage-el6]# docker info
Containers: 3
Images: 37
Server Version: 1.9.1-fc23
Storage Driver: devicemapper
Pool Name: docker-253:0-567770-pool
Pool Blocksize: 65.54 kB
Base Device Size: 21.47 GB
Backing Filesystem:
Data file: /dev/docker/containers
Metadata file: /dev/docker/metadata
Data Space Used: 2.893 GB
Data Space Total: 192.2 GB
Data Space Available: 189.3 GB
Metadata Space Used: 4.415 MB
Metadata Space Total: 17.05 GB
Metadata Space Available: 17.04 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.109 (2015-09-22)
Execution Driver: native-0.2
Logging Driver: journald
Kernel Version: 4.2.7-300.fc23.x86_64
Operating System: Fedora 23 (Twenty Three)
CPUs: 4
Total Memory: 3.86 GiB
Name: XXXXXXXXXXXX
ID: XXXXXXXXXX
docker inspect <CONTAINERID>
[
{
"Id": "<CONTAINERID>",
"Created": "2016-01-13T19:34:32.062398984Z",
"Path": "/usr/sbin/init",
"Args": [],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1918,
"ExitCode": 0,
"Error": "",
"StartedAt": "2016-01-13T19:34:39.579629283Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "7a21760c41c742ab3cfaacb4b0fb7a03406781cacd8024ad28526c5ba97839cd",
"ResolvConfPath": "/var/lib/docker/containers/fd9b62d7154471495f3fd6816e7c36399a86a9fb44ada91fa90b90228974940b/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/fd9b62d7154471495f3fd6816e7c36399a86a9fb44ada91fa90b90228974940b/hostname",
"HostsPath": "/var/lib/docker/containers/fd9b62d7154471495f3fd6816e7c36399a86a9fb44ada91fa90b90228974940b/hosts",
"LogPath": "",
"Name": "/pensive_mcclintock",
"RestartCount": 0,
"Driver": "devicemapper",
"ExecDriver": "native-0.2",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": [
"31d846f24e228729b2d5c3f9a38e324ab6cc23651b666f0dd24b9666161ac54d"
],
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LxcConf": [],
"Memory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"KernelMemory": 0,
"CpuShares": 0,
"CpuPeriod": 0,
"CpusetCpus": "",
"CpusetMems": "",
"CpuQuota": 0,
"BlkioWeight": 0,
"OomKillDisable": false,
"MemorySwappiness": -1,
"Privileged": true,
"PortBindings": {},
"Links": null,
"PublishAllPorts": false,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"VolumesFrom": null,
"Devices": [],
"NetworkMode": "default",
"IpcMode": "",
"PidMode": "",
"UTSMode": "",
"CapAdd": null,
"CapDrop": null,
"GroupAdd": null,
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"SecurityOpt": null,
"ReadonlyRootfs": false,
"Ulimits": null,
"LogConfig": {
"Type": "journald",
"Config": {}
},
"CgroupParent": "",
"ConsoleSize": [
0,
0
],
"VolumeDriver": "",
"ShmSize": 67108864
},
"GraphDriver": {
"Name": "devicemapper",
"Data": {
"DeviceId": "290",
"DeviceName": "docker-253:0-567770-fd9b62d7154471495f3fd6816e7c36399a86a9fb44ada91fa90b90228974940b",
"DeviceSize": "21474836480"
}
},
"Mounts": [
{
"Name": "717d5c9b85ef494b7bd0520f2d1a8425a96d99c08d00b13f742161b604f4ce73",
"Source": "/var/lib/docker/volumes/717d5c9b85ef494b7bd0520f2d1a8425a96d99c08d00b13f742161b604f4ce73/_data",
"Destination": "/sys/fs/cgroup",
"Driver": "local",
"Mode": "",
"RW": true
},
{
"Name": "43491f02e24e460d41b91b1becca20655c173a47d6f5d86ad1c1ecaee0af8c56",
"Source": "/var/lib/docker/volumes/43491f02e24e460d41b91b1becca20655c173a47d6f5d86ad1c1ecaee0af8c56/_data",
"Destination": "/run",
"Driver": "local",
"Mode": "",
"RW": true
}
],
"Config": {
"Hostname": "fd9b62d71544",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": true,
"AttachStderr": true,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": null,
"Cmd": null,
"Image": "docker.io/centos:centos7",
"Volumes": {
"/run": {},
"/sys/fs/cgroup": {}
},
"WorkingDir": "",
"Entrypoint": [
"/usr/sbin/init"
],
"OnBuild": null,
"Labels": {
"license": "GPLv2",
"name": "CentOS Base Image",
"vendor": "CentOS"
},
"StopSignal": "SIGTERM"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "95d6b4432e7eccd68e78d682304c4edb0824c54c4399b681fdc1efe57a8317e6",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/95d6b4432e7e",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "363bdca9844a6c0b0f153e3ac4ed246c8eee368d082ddef597fbd947dc890998",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:02",
"Networks": {
"bridge": {
"EndpointID": "363bdca9844a6c0b0f153e3ac4ed246c8eee368d082ddef597fbd947dc890998",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:02"
}
}
}
}
]
Systemd rpm version:
systemd-219-19.el7.x86_64
Thanks,
-Cesar
Looks this is a systemd issue. I am able to run your find test successfully when I change entrypoint to bash rather than init. When systemd is running inside the container and a bind mount is issued, I find the following message surfaces the problem: # journalctl |grep systemd |grep mount Jan 14 14:25:46 5d55a45afec8 systemd[1]: Unit etc-hostname.mount is bound to inactive unit dev-md2.device. Stopping, too. Jan 14 14:25:46 5d55a45afec8 systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point. Jan 14 14:25:46 5d55a45afec8 systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point. Jan 14 14:25:46 5d55a45afec8 systemd[1]: Starting Remount Root and Kernel File Systems... Jan 14 14:25:46 5d55a45afec8 systemd[1]: Unit etc-hostname.mount is bound to inactive unit dev-md2.device. Stopping, too. Jan 14 14:25:46 5d55a45afec8 systemd[1]: Unit etc-hosts.mount is bound to inactive unit dev-md2.device. Stopping, too. Jan 14 14:25:46 5d55a45afec8 systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point. Jan 14 14:25:46 5d55a45afec8 systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point. Jan 14 14:25:46 5d55a45afec8 systemd[1]: Starting Remount Root and Kernel File Systems... Jan 14 14:25:46 5d55a45afec8 systemd-remount-fs[24]: mount: can't find UUID=1c74ece3-3844-44a6-a1b6-f8defa8f03fd Jan 14 14:25:46 5d55a45afec8 systemd-remount-fs[24]: /bin/mount for / exited with exit status 1. Jan 14 14:25:46 5d55a45afec8 systemd[1]: systemd-remount-fs.service: main process exited, code=exited, status=1/FAILURE Jan 14 14:25:46 5d55a45afec8 systemd[1]: Failed to start Remount Root and Kernel File Systems. Jan 14 14:25:46 5d55a45afec8 systemd[1]: Unit systemd-remount-fs.service entered failed state. Jan 14 14:25:46 5d55a45afec8 systemd[1]: systemd-remount-fs.service failed. Jan 14 14:26:18 5d55a45afec8 systemd[1]: Unit tmp-test.mount is bound to inactive unit dev-mapper-docker\x2d9:2\x2d9568392\x2d5d55a45afec8656c43dc84753ac62e00f7860e5a88b30c344713469a968e6b1d.device. Stopping, too. Jan 14 14:26:18 5d55a45afec8 systemd[1]: Unmounting /tmp/test... Jan 14 14:26:18 5d55a45afec8 systemd[1]: Unmounted /tmp/test. @All, we tried the fix mentioned @ https://bugzilla.redhat.com/show_bug.cgi?id=1294459#c7 and it seems that it resolves our issues. (In reply to Humble Chirammal from comment #11) > @All, we tried the fix mentioned @ > https://bugzilla.redhat.com/show_bug.cgi?id=1294459#c7 and it seems that it > resolves our issues. Unfortunately we still hit some part of the issue mentioned in this bugzilla. That said, when spawning containers from image which has systemd fixes mentioned in c#11, the bind mounted volumes are getting unmounted from the atomic hosts. however it noticed that, the volumes are mounted inside the container. I will update the bugzilla with the details soon. (In reply to Humble Chirammal from comment #12) > (In reply to Humble Chirammal from comment #11) > > @All, we tried the fix mentioned @ > > https://bugzilla.redhat.com/show_bug.cgi?id=1294459#c7 and it seems that it > > resolves our issues. > > Unfortunately we still hit some part of the issue mentioned in this > bugzilla. That said, when spawning containers from image which has systemd > fixes mentioned in c#11, the bind mounted volumes are getting unmounted > from the atomic hosts. however it noticed that, the volumes are mounted > inside the container. I will update the bugzilla with the details soon. When we are bind mounting volume into the rhgs container from atomic host, The issues we face are 1) LV unmount happening in the atomic host automatically https://bugzilla.redhat.com/show_bug.cgi?id=1294776 2) docker stop/kill hung https://bugzilla.redhat.com/show_bug.cgi?id=1293667 we tried remounting the LV's before doing docker stop, Still docker stop hangs. we had to restart docker service to run docker commands. The complete setup and issue is been recorded here: https://bugzilla.redhat.com/show_bug.cgi?id=1293667#c14 -bash-4.2# rpm -qa | grep docker docker-1.8.2-10.el7.x86_64 docker-selinux-1.8.2-10.el7.x86_64 python-docker-py-1.4.0-118.el7.noarch -bash-4.2# rpm -qa | grep systemd systemd-sysv-219-19.el7.x86_64 systemd-219-19.el7.x86_64 systemd-libs-219-19.el7.x86_64 Could we get an update on this bug, Are you still seeing these issues? I am going to assume this is fixed in the latest versions, since we never heard about it again. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |