Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
+++ This bug was initially created as a clone of Bug #1498281 +++
# docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker \
docker.io/fedora:latest findmnt -R -n --list / |grep /overlay2
/rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 /dev/mapper/atomicos-docker--root--lv[/overlay2] xfs rw,relatime,seclabel,attr2,inode64,noquota
# docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker \
ls -l /rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2
[lots of hashes]
Environment: docker-1.13.1-22.gitb5e3294.fc26.x86_64 on Fedora Atomic Host 26
--- Additional comment from Daniel Walsh on 2017-10-03 21:45:22 EDT ---
Do you have oci-umount installed?
--- Additional comment from Ed Santiago on 2017-10-03 22:02:57 EDT ---
oci-umount-2.0.0-2.gitf90b64c.fc26.x86_64
This is on Fedora Atomic Host with `atomic host upgrade` as of 2017-10-03 PM.
# atomic host status
State: idle
Deployments:
? fedora-atomic:fedora/26/x86_64/atomic-host
Version: 26.131 (2017-09-19 22:29:04)
Commit: 98088cb6ed2a4b3f7e4e7bf6d34f9e137c296bc43640b4c1967631f22fe1802f
GPGSignature: Valid signature by E641850B77DF435378D1D7E2812A6B4B64DAB85D
fedora-atomic:fedora/26/x86_64/atomic-host
Version: 26.83 (2017-07-05 13:04:12)
Commit: 2e8d8dab1c794ff61ba79e60652a687cd1b3c43ad97f0aac38fd271553e58ee2
GPGSignature: Valid signature by E641850B77DF435378D1D7E2812A6B4B64DAB85D
I greatly regret that I am going offline now and will not be able to reply again until, at the hopeful earliest, Friday.
--- Additional comment from Daniel Walsh on 2017-10-03 22:10:30 EDT ---
cat /etc/oci-umount.conf
--- Additional comment from Vivek Goyal on 2017-10-04 08:43:09 EDT ---
So there are two paths inside container. /var/lib/docker/overlay2 got unmounted but /rootfs/....../overlay2 did not.
I think oci-umount logic expected /rootfs/var/lib/docker/overlay2 to be present because atomic does so much of magic with mounts, its not there. Instead, "/rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2", is there.
But this sounds like an atomic problem. If I do "-v /:/rootfs", shouldn't I see a mount point on /rootfs/var/lib/docker/overlay2 inside container?
--- Additional comment from Vivek Goyal on 2017-10-04 08:43:39 EDT ---
Ed, please also paste journalctl logs which are prefixed with umounthook.
--- Additional comment from Daniel Walsh on 2017-10-04 19:48:51 EDT ---
I think the /rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 is unrelated and is not a leaked mount point into this container. It is a mount point on the host, but unrelated to the container.
--- Additional comment from Ed Santiago on 2017-10-06 11:47:01 EDT ---
/etc/oci-umount.conf:
# This contains a list of paths on host which will be unmounted inside
# container. (If they are mounted inside container).
# If there is a "/*" at the end, that means only mounts underneath that
# mounts (submounts) will be unmounted but top level mount will remain
# in place.
/var/lib/docker/overlay2
/var/lib/docker/overlay
/var/lib/docker/devicemapper
/var/lib/docker/containers/*
/var/lib/docker-latest/overlay2
/var/lib/docker-latest/overlay
/var/lib/docker-latest/devicemapper
/var/lib/docker-latest/containers/*
/var/lib/containers/storage/lvm
/var/lib/containers/storage/devicemapper
/var/lib/containers/storage/overlay
/var/run/containers/storage
--- Additional comment from Ed Santiago on 2017-10-06 11:51 EDT ---
Attached: journalctl -f | grep umounthook
command run in other window: docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker docker.io/fedora:latest findmnt -R -n --list / |grep /overlay2
--- Additional comment from Daniel Walsh on 2018-01-22 15:51:59 EST ---
Ed is this still the case with the latest oci-umount?
--- Additional comment from Ed Santiago on 2018-01-22 16:43:06 EST ---
Dan: yes, it looks like it:
# docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker docker.io/fedora:latest findmnt -R -n --list / |grep /overlay2
/ overlay overlay rw,relatime,context="system_u:object_r:container_file_t:s0:c9,c93",lowerdir=/var/lib/docker/overlay2/l/KNWJRV376GNMPNXU2SMUUJGH2F:/var/lib/docker/overlay2/l/THD7KDPGXPUUNNDWWYLPZXM2TJ,upperdir=/var/lib/docker/overlay2/11f60c6ee54e427f4cfd20e30c6a24fb38cdc0e1e745fcf5bb2196344b73fb2d/diff,workdir=/var/lib/docker/overlay2/11f60c6ee54e427f4cfd20e30c6a24fb38cdc0e1e745fcf5bb2196344b73fb2d/work
/rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 /dev/mapper/atomicos-root[/ostree/deploy/fedora-atomic/var/lib/docker/overlay2] xfs rw,relatime,seclabel,attr2,inode64,noquota
# docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker docker.io/fedora:latest ls -l /rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2
ls: cannot access '/rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2/backingFsBlockDev': Permission denied
total 0
drwx------. 5 root root 69 Jan 22 21:37 5ddb64baba16bf80ed436f40b46c3e7e0afaf195fb41c397290249698f72e311
drwx------. 5 root root 69 Jan 22 21:37 5ddb64baba16bf80ed436f40b46c3e7e0afaf195fb41c397290249698f72e311-init
b?????????? ? ? ? ? ? backingFsBlockDev
drwx------. 3 root root 30 Jan 22 21:35 d254fd08ae8efabbfcd5d5c5f4e98457b15dd921466758f2a13cd0eb4502c8b3
drwx------. 2 root root 108 Jan 22 21:37 l
host details:
# atomic host status
State: idle
Deployments:
? fedora-atomic:fedora/27/x86_64/atomic-host
Version: 27.61 (2018-01-17 15:52:47)
Commit: 772ab185b0752b0d6bc8b2096d08955660d80ed95579e13e136e6a54e3559ca9
GPGSignature: Valid signature by 860E19B0AFA800A1751881A6F55E7430F5282EE4
rpms:
docker-1.13.1-44.git584d391.fc27.x86_64
oci-umount-2.3.2-1.git3025b19.fc27.x86_64
oci-umount.conf :
# This contains a list of paths on host which will be unmounted inside
# container. (If they are mounted inside container).
# If there is a "/*" at the end, that means only mounts underneath that
# mounts (submounts) will be unmounted but top level mount will remain
# in place.
/var/lib/docker/overlay2
/var/lib/docker/overlay
/var/lib/docker/devicemapper
/var/lib/docker/containers/*
/var/lib/docker-latest/overlay2
/var/lib/docker-latest/overlay
/var/lib/docker-latest/devicemapper
/var/lib/docker-latest/containers/*
--- Additional comment from Daniel Walsh on 2018-01-24 04:56:20 EST ---
Vivek thoughts?
--- Additional comment from Vivek Goyal on 2018-01-25 13:08:26 EST ---
I think it is a problem where source mount is visible on host in multiple paths and we don't have any mechanism to deal with it. For example, on a regular fedora host, do following.
# mkdir /root/foo/foo2
# mount --rbind /var/lib/ /root/foo/foo2
# docker run -ti -v /:/rootfs fedora bash
Now inside the container you will still see the overlay2 mount through /rootfs/foo/foo2/docker/ovlerlay2
oci-umount has already taken out two mounts. One at /var/lib/docker/overlay2
and another one at /rootfs/var/lib/docker/overlay2. But it does not know anything about source path on host "/root/foo/foo2/docker/overlay2" and when that path leaks inside container due to "-v /:/rootfs", oci-umount can't do anything about it.
To get rid of it, oci-umount will have to be told about it explicitly. So put something like this in /etc/oci-umount.conf
/root/foo/foo2/docker/overlay2
Similarly for atomic case, if we put something like this in oci-umount.conf, that should take care of it.
/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2
Ed, can you try modifying /etc/oci-umount.conf and put above and see if it helps.
--- Additional comment from Ed Santiago on 2018-01-25 15:54:03 EST ---
Yes, it helps:
# docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker docker.io/fedora:latest findmnt -R -n --list / |grep /overlay2
/ overlay overlay rw,relatime,context="system_u:object_r:container_file_t:s0:c297,c565",lowerdir=/var/lib/docker/overlay2/l/MU22NS3P366FKOLXVPQSWEDPEH:/var/lib/docker/overlay2/l/EQG7KJX7LIONH4KNW4W6QO4CYC,upperdir=/var/lib/docker/overlay2/1f209110ec4de4a7a504f278b2c6c5fb47650909b36ad9be87f488d89ec00197/diff,workdir=/var/lib/docker/overlay2/1f209110ec4de4a7a504f278b2c6c5fb47650909b36ad9be87f488d89ec00197/work
But as you imply, this is just a workaround. The real issue, as I see it, is that oci-umount works on a blacklist basis ("everything is OK except these few") -- there will always be ways to bypass that, almost always unintentionally.
--- Additional comment from Vivek Goyal on 2018-01-25 16:05:33 EST ---
I think that's the right fix. (its not a workaround).
No, oci-umount does not work on blacklist basis. It just works on the list of paths (source paths on host) in /etc/oci-umount.conf and if that path is mounted inside container, it unmounts that. It does not care about other mounts other than listed in /etc/oci-umount.conf.
/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 is not same as /var/lib/docker/overlay2. So we will have to explicitly specify it in /etc/oci-umount.conf.
I think atomic kickstart might have to be modified for this.
Colin, WDYT.
--- Additional comment from Colin Walters on 2018-01-26 02:00:29 EST ---
The /sysroot thing seems like a plain ostree bug - offhand, I think we should be making /sysroot slave and not shared. Let me do some testing.
--- Additional comment from Colin Walters on 2018-01-29 12:58:40 EST ---
Upstream PR: https://github.com/ostreedev/ostree/pull/1438
--- Additional comment from Daniel Walsh on 2018-01-30 10:55:40 EST ---
Since this is fixed upstream, we don't need to fix this in oci-umount? Is that correct? Do we have this issue in RHEL 7?
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHEA-2018:1067