RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1568450 - /var/lib/docker/overlay2 unmounted when docker stopped in 1.13.1-62
Summary: /var/lib/docker/overlay2 unmounted when docker stopped in 1.13.1-62
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.4
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Daniel Walsh
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-17 14:18 UTC by Mike Fiedler
Modified: 2020-06-09 21:39 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-09 21:39:50 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Mike Fiedler 2018-04-17 14:18:39 UTC 
Description of problem:

On RHEL 7.4.5 and docker 1.13.1-62, when docker is stopped (no pods running), the /var/lib/docker/overlay2 filesystem is unmounted.

If /var/lib/docker/overlay2 is removed from oci-umount.conf the problem does not occur.  My understanding is that oci-umount.conf is to control mounts within pods, not OS mounts.

This problem does not occur with docker 1.12.6-68

Version-Release number of selected component (if applicable):  1.13.1-62


How reproducible: Always


Steps to Reproduce:
1.  Install docker 1.13.1-62.
2.  Create an XFS filesystem for /var/lib/docker/overlay2.  configure docker to use the overlay2 storage driver
3.  df -h to verify the fs is mounted
4.  systemctl stop docker  (or restart)

Actual results:

/var/lib/docker/overlay2 is unmounted


Expected results:

/var/lib/docker/overlay2 fs survives a docker restart


Additional info:
Comment 2 Daniel Walsh 2018-04-18 10:43:37 UTC 
Why not configure your XFS to mount at /var/lib/docker?  docker/oci-umount are both mounting and umounting the driver directories.  Their are several reasons for this but mainly leaked file descriptors.  You could also remove the /var/lib/docker/ovleray2 directory from /etc/oci-umounts.conf.
Comment 3 Mike Fiedler 2018-04-18 11:16:45 UTC 
Both suggestions in comment 2 are valid - thanks.  I opened the bz to track the behavior change.   Setting severity to medium - is this just a doc issue?
Comment 4 Daniel Walsh 2018-04-18 12:16:17 UTC 
Sure we can make this a docs issue.
Comment 5 Vivek Goyal 2018-04-24 14:20:20 UTC 
Yes, don't mount on /var/lib/docker/overlay2. I suspect docker is the one which unmounts that mount point when it stops (And not oci-umount). Docker creates a private mount there and unmounts when stopping. If mount point is already there, I think then a new mount point is not created but unmounted on stop. Not sure, but that's my current thought process.
Comment 6 Mike Fiedler 2018-04-24 14:38:12 UTC 
I agree it is likely not oci-umount.  Removing /var/lib/docker/overlay2 from /etc/oci-umount.conf did not help.  We'll change our mount to /var/lib/docker
Comment 7 Vivek Goyal 2018-04-24 14:54:01 UTC 
Mike, we already talked about it on IRC. I am putting same information here so that anybody looking later can find it.

- openshift folks, try to make use of docker-storage-setup to setup a separate volume and put xfs on top of that volume and mount it on /var/lib/docker. 

- You can use following /etc/sysconfig/docker-storage-setup for that.

DEVS="/dev/vdb"
VG=docker-vg
CONTAINER_ROOT_LV_MOUNT_PATH="/var/lib/docker/"
CONTAINER_ROOT_LV_NAME="docker-root-lv"
STORAGE_DRIVER=overlay2

This will create a volume group "docker-vg" from disk /dev/vdb, create a logical volme docker-root-lv from volume group, create xfs filesystem on this logical volume and mount on /var/lib/docker. And over restart, just mounting step happens.

- From command line do following.

 - systemctl stop docker
 - atomic storage reset
 - Edit /etc/sysconfig/docker-storage-setup appropriately
 - systemctl start docker


And you should be all set.
Comment 8 Vivek Goyal 2018-04-24 15:58:40 UTC 
I looked at config of another node I think used by openstack folks. I noticed one more things which probably make sense.

CONTAINER_ROOT_LV_SIZE=100%FREE

This specifies use 100% of free space in VG for the logical volume. By default it uses 40%of free space. So config should look something as follows.

DEVS="/dev/vdb"
VG=docker-vg
CONTAINER_ROOT_LV_MOUNT_PATH="/var/lib/docker/"
CONTAINER_ROOT_LV_NAME="docker-root-lv"
STORAGE_DRIVER=overlay2
CONTAINER_ROOT_LV_SIZE=100%FREE
Comment 9 Vivek Goyal 2018-04-24 16:02:05 UTC 
This recommendation is assuming that you don't want to use root filesystem of the host to storage containers and images and want to use a separate disk. If you want to use rootfs of host, then all you need is.

STORAGE_DRIVER=overlay2

So first decide what are your requirements and then choose the container-storage-setup accordingly.
Comment 10 Tom Sweeney 2020-06-09 21:39:50 UTC 
We have no plans to ship another version of Docker at this time. RHEL7 is in final support stages where only security fixes will get released.  Customers should move to use Podman which is available starting in RHEL 7.6.
Note You need to log in before you can comment on or make changes to this bug.