Bug 1412728

Summary: Bind mount on /var/log seems to be over shadowed
Product: Red Hat Enterprise Linux 7 Reporter: Mohamed Ashiq <mliyazud>
Component: oci-systemd-hookAssignee: Mrunal Patel <mpatel>
Status: CLOSED ERRATA QA Contact: Martin Jenner <mjenner>
Severity: urgent Docs Contact:
Priority: high    
Version: 7.3CC: amurdaca, cww, ddarrah, dwalsh, gouyang, hchiramm, lsm5, mliyazud, mpatel, pprakash, rcyriac
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: oci-systemd-hook-1:0.1.4-9.git671c428.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-17 20:46:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1412281    

Description Mohamed Ashiq 2017-01-12 16:10:50 UTC
Description of problem:
We have a systemd container on which we have to have a bind mount on the '/var/log/<something>'. This worked fine before this release. This bind mount is strict requirement for us because of which we are hitting issue in our setups. 

Version-Release number of selected component (if applicable):
# rpm -qa | grep docker
cockpit-docker-126-1.el7.x86_64
docker-common-1.12.5-14.el7.x86_64
docker-1.12.5-14.el7.x86_64
docker-client-1.12.5-14.el7.x86_64
docker-rhel-push-plugin-1.12.5-14.el7.x86_64

# rpm -qa | grep systemd
systemd-219-30.el7_3.7.x86_64
systemd-sysv-219-30.el7_3.7.x86_64
oci-systemd-hook-0.1.4-8.git45455fe.el7.x86_64
systemd-libs-219-30.el7_3.7.x86_64

# rpm -qa | grep systemd
systemd-libs-219-30.el7_3.6.x86_64
systemd-219-30.el7_3.6.x86_64

How reproducible:
Always

Steps to Reproduce:
# docker run -d -v /var/log/something:/var/log/something:z rhel /usr/sbin/init
b8a202e69e364f0cee13a7127bc97a7ad9f55e5827e5462b5bc4ed09b1f12f74
# docker exec -it b8a202e69e364f0cee13a7127bc97a7ad9f55e5827e5462b5bc4ed09b1f12f74 bash
/]# df -h
Filesystem                                                                                         Size  Used Avail Use% Mounted on
/dev/mapper/docker-8:17-67158286-1c93270178ee4a8845a0b69c0729a516ffd9dc712d0bf7c2cb160021f02c94b0   10G  247M  9.8G   3% /
tmpfs                                                                                               24G     0   24G   0% /dev
tmpfs                                                                                               24G     0   24G   0% /sys/fs/cgroup
/dev/sdb1                                                                                           40G  1.2G   39G   3% /etc/hosts
shm                                                                                                 64M     0   64M   0% /dev/shm
tmpfs                                                                                               64M  236K   64M   1% /run
tmpfs                                                                                              4.0E     0  4.0E   0% /tmp
tmpfs                                                                                              4.0E  8.0K  4.0E   1% /var/log
/]# ls /var/log/
btmp     journal/ wtmp

/]# mount       
/dev/mapper/docker-8:17-67158286-1c93270178ee4a8845a0b69c0729a516ffd9dc712d0bf7c2cb160021f02c94b0 on / type xfs (rw,relatime,seclabel,nouuid,attr2,inode64,sunit=1024,swidth=1024,noquota)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,seclabel,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_prio,net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpuacct,cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
/dev/sdb1 on /etc/resolv.conf type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/sdb1 on /etc/hostname type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/sdb1 on /etc/hosts type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,size=65536k)
/dev/sdb1 on /run/secrets type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/sdb1 on /var/log/something type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,size=65536k,mode=755)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=4503599627370496k)
tmpfs on /var/log type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=4503599627370496k)
/dev/sdb1 on /var/log/journal/552de008be2c0a1364cbacfc32ef526f type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,relatime,cpuset)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,relatime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,relatime,freezer)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
configfs on /sys/kernel/config type configfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)


Please let me know if you need more information.

Comment 1 Daniel Walsh 2017-01-12 16:12:55 UTC
What version of oci-systemd-hook do you have installed

rpm -q oci-systemd-hook

Comment 2 Mohamed Ashiq 2017-01-12 16:16:25 UTC
# rpm -qa | grep oci
oci-register-machine-0-1.11.gitdd0daef.el7.x86_64
oci-systemd-hook-0.1.4-8.git45455fe.el7.x86_64


In the description First systemd version is on the host and second one is on container.

Comment 3 Daniel Walsh 2017-01-12 16:36:16 UTC
Mrunal, we are overmounting the directories on top of existing volume mounts and on /run/secrets.

First question?  Do you remember why we are mounting a tmpfs on /var/log?

When we mount over /run we should probably tar up the contents from the directory and put them into the tmpfs of at least mv any mount points off of /run into the newly mounted tmpfs.

# docker run -ti -v /var/log/dan:/var/log/dan:z fedora mount | grep /var/log
/dev/sda2 on /var/log/dan type ext4 (rw,relatime,seclabel,data=ordered)
tmpfs on /var/log type tmpfs (rw,nosuid,nodev,relatime,context="system_u:object_r:container_file_t:s0:c118,c249")
/dev/sda2 on /var/log/journal/184b22d455aafb6c9d56ce1f79cf3d20 type ext4 (rw,relatime,seclabel,data=ordered)

I can work on a fix, but I want  your opinion.

Comment 4 Mrunal Patel 2017-01-12 17:53:21 UTC
Dan, the /var/log tmpfs was for journald logs. I think we can do the same dance of MS_MOVE that we do for /run mounts. We move mounts temporarily using MS_MOVE and then put them back in place for anything over /run. We can do the same for mounts specified by the user over /var/log.

Comment 6 Guohua Ouyang 2017-01-13 02:54:10 UTC
tested oci-systemd-hook-0.1.4-9.git671c428.el7.x86_64.rpm, the fix works.

1. Reproduced the bug 
# mkdir /var/log/test
# docker run -d -v /var/log/test:/var/log/test:z rhel /usr/sbin/init
44f4d3a237c94b6f0b870f972f9adf638407827103283a19fccb55433495fd7b
# docker exec -it 44f4d3a237 bash
# ls /var/log
btmp  journal  wtmp
# docker stop 44f4
44f4
# docker rm 44f4
44f4

2. 
# rpm -Uvh oci-systemd-hook-0.1.4-9.git671c428.el7.x86_64.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:oci-systemd-hook-1:0.1.4-9.git671################################# [ 50%]
Cleaning up / removing...
   2:oci-systemd-hook-1:0.1.4-8.git454################################# [100%]

3. 
# docker run -d -v /var/log/test:/var/log/test:z rhel /usr/sbin/init
2c46308f2981e72fd19378da393e8719faaee85cdeec597377bb73b13c5b8133
# docker exec -it 2c46308 bash
# ls /var/log
btmp  journal  test  wtmp

The /var/log/test dir is there.

Comment 8 Humble Chirammal 2017-01-13 06:46:09 UTC
Thanks Dan for your quick help on this!! We are also validating the fix from our side and will update this bug accordingly.

Brew link : https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=12348885

Comment 9 Daniel Walsh 2017-01-13 12:31:55 UTC
Awesome.

Comment 10 Humble Chirammal 2017-01-13 14:59:02 UTC
Good news!! Gluster Container deployment is working as expected and we confirm that, the reported issue at our end is fixed with above mentioned build. Once again thanks a lot for quick help on this, much appreciated.

Comment 13 errata-xmlrpc 2017-01-17 20:46:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0117.html

Comment 14 Red Hat Bugzilla 2023-09-14 03:37:20 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days