Bug 1568450

Summary: /var/lib/docker/overlay2 unmounted when docker stopped in 1.13.1-62
Product: Red Hat Enterprise Linux 7 Reporter: Mike Fiedler <mifiedle>
Component: dockerAssignee: Daniel Walsh <dwalsh>
Status: CLOSED WONTFIX QA Contact: atomic-bugs <atomic-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.4CC: amurdaca, gkeegan, lsm5, pasik, sdodson, tsweeney, vgoyal, wabouham
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-06-09 21:39:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mike Fiedler 2018-04-17 14:18:39 UTC
Description of problem:

On RHEL 7.4.5 and docker 1.13.1-62, when docker is stopped (no pods running), the /var/lib/docker/overlay2 filesystem is unmounted.

If /var/lib/docker/overlay2 is removed from oci-umount.conf the problem does not occur.  My understanding is that oci-umount.conf is to control mounts within pods, not OS mounts.

This problem does not occur with docker 1.12.6-68

Version-Release number of selected component (if applicable):  1.13.1-62


How reproducible: Always


Steps to Reproduce:
1.  Install docker 1.13.1-62.
2.  Create an XFS filesystem for /var/lib/docker/overlay2.  configure docker to use the overlay2 storage driver
3.  df -h to verify the fs is mounted
4.  systemctl stop docker  (or restart)

Actual results:

/var/lib/docker/overlay2 is unmounted


Expected results:

/var/lib/docker/overlay2 fs survives a docker restart


Additional info:

Comment 2 Daniel Walsh 2018-04-18 10:43:37 UTC
Why not configure your XFS to mount at /var/lib/docker?  docker/oci-umount are both mounting and umounting the driver directories.  Their are several reasons for this but mainly leaked file descriptors.  You could also remove the /var/lib/docker/ovleray2 directory from /etc/oci-umounts.conf.

Comment 3 Mike Fiedler 2018-04-18 11:16:45 UTC
Both suggestions in comment 2 are valid - thanks.  I opened the bz to track the behavior change.   Setting severity to medium - is this just a doc issue?

Comment 4 Daniel Walsh 2018-04-18 12:16:17 UTC
Sure we can make this a docs issue.

Comment 5 Vivek Goyal 2018-04-24 14:20:20 UTC
Yes, don't mount on /var/lib/docker/overlay2. I suspect docker is the one which unmounts that mount point when it stops (And not oci-umount). Docker creates a private mount there and unmounts when stopping. If mount point is already there, I think then a new mount point is not created but unmounted on stop. Not sure, but that's my current thought process.

Comment 6 Mike Fiedler 2018-04-24 14:38:12 UTC
I agree it is likely not oci-umount.  Removing /var/lib/docker/overlay2 from /etc/oci-umount.conf did not help.  We'll change our mount to /var/lib/docker

Comment 7 Vivek Goyal 2018-04-24 14:54:01 UTC
Mike, we already talked about it on IRC. I am putting same information here so that anybody looking later can find it.

- openshift folks, try to make use of docker-storage-setup to setup a separate volume and put xfs on top of that volume and mount it on /var/lib/docker. 

- You can use following /etc/sysconfig/docker-storage-setup for that.

DEVS="/dev/vdb"
VG=docker-vg
CONTAINER_ROOT_LV_MOUNT_PATH="/var/lib/docker/"
CONTAINER_ROOT_LV_NAME="docker-root-lv"
STORAGE_DRIVER=overlay2

This will create a volume group "docker-vg" from disk /dev/vdb, create a logical volme docker-root-lv from volume group, create xfs filesystem on this logical volume and mount on /var/lib/docker. And over restart, just mounting step happens.

- From command line do following.

 - systemctl stop docker
 - atomic storage reset
 - Edit /etc/sysconfig/docker-storage-setup appropriately
 - systemctl start docker


And you should be all set.

Comment 8 Vivek Goyal 2018-04-24 15:58:40 UTC
I looked at config of another node I think used by openstack folks. I noticed one more things which probably make sense.

CONTAINER_ROOT_LV_SIZE=100%FREE

This specifies use 100% of free space in VG for the logical volume. By default it uses 40%of free space. So config should look something as follows.

DEVS="/dev/vdb"
VG=docker-vg
CONTAINER_ROOT_LV_MOUNT_PATH="/var/lib/docker/"
CONTAINER_ROOT_LV_NAME="docker-root-lv"
STORAGE_DRIVER=overlay2
CONTAINER_ROOT_LV_SIZE=100%FREE

Comment 9 Vivek Goyal 2018-04-24 16:02:05 UTC
This recommendation is assuming that you don't want to use root filesystem of the host to storage containers and images and want to use a separate disk. If you want to use rootfs of host, then all you need is.

STORAGE_DRIVER=overlay2

So first decide what are your requirements and then choose the container-storage-setup accordingly.

Comment 10 Tom Sweeney 2020-06-09 21:39:50 UTC
We have no plans to ship another version of Docker at this time. RHEL7 is in final support stages where only security fixes will get released.  Customers should move to use Podman which is available starting in RHEL 7.6.