Bug 1441737 - [runc spec] Enable kernel sysctl knob /proc/sys/fs/may_detach_mounts
Summary: [runc spec] Enable kernel sysctl knob /proc/sys/fs/may_detach_mounts
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: runc
Version: 7.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Lokesh Mandvekar
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1468249 1441743 1542672
TreeView+ depends on / blocked
 
Reported: 2017-04-12 15:11 UTC by Vivek Goyal
Modified: 2018-10-13 11:53 UTC (History)
8 users (show)

Fixed In Version: runc-1.0.0-12.1.gitf8ce01d.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1468249 (view as bug list)
Environment:
Last Closed: 2017-08-02 00:17:16 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2350 normal SHIPPED_LIVE runc bug fix and enhancement update 2017-08-08 22:52:27 UTC

Description Vivek Goyal 2017-04-12 15:11:55 UTC
Description of problem:

RHEL 7.4 kernel has introduced a new sysctl knob to control kernel behavior. This is called /proc/sys/fs/may_detach_mounts. This knob is set to value 0 by default. Container run times (docker and others) need the new behavior and
want it to be set to 1. 

So modify runc package to drop a file say /usr/lib/sysctl.d/99-docker.conf. Contents of this file can be say following.

fs.may_detach_mounts=1

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Vivek Goyal 2017-04-12 15:16:29 UTC
We need this change so that we can run docker daemon in host mount namespace. And that will enable shared volume feature of docker where volumes mounted by container can propagate to host (if user has configured it right).

Comment 4 Daniel Walsh 2017-04-12 16:14:04 UTC
What happens if this file gets placed on a system without the kernel mode?  I think it just gets ignored correct?  Ie we install the runc package on a RHEL7.3 OS>

Comment 5 Vivek Goyal 2017-04-12 17:26:19 UTC
I think that it will be ignored. IOW, I think systemd will try to write to this file but it will not be present. I am assuming that systemd will continue to write rest of the knobs.

I tried it on fedora kernel and I see following message in logs.

Apr 12 13:25:28 vm7-f25 systemd-sysctl[22835]: Couldn't write '1' to 'fs/may_detach_mounts', ignoring: No such file or directory

Comment 6 Daniel Walsh 2017-04-13 18:46:03 UTC
Lokesh lets drop  /usr/lib/sysctl.d/99-containers.conf  with this flag, for rhel7.4.

Comment 7 Ed Santiago 2017-06-08 15:18:48 UTC
This RPM install is not behaving as I had expected: the may_detach_mounts option is not taking effect until after a reboot. I believe the specfile is missing a %sysctl_apply directive:

    https://fedoraproject.org/wiki/Packaging:Guidelines#binfmt.d.2C_sysctl.d_and_tmpfiles.d

Comment 8 Vivek Goyal 2017-06-08 18:42:36 UTC
Good point Ed. 

Lokesh, we probably need to fix it.

Comment 11 Vivek Goyal 2017-07-06 13:02:39 UTC
This does not help with docker install. Docker package seems to ship its own runc and does not have dependency to install runc package. That means after installing docker, /proc/sys/fs/may_detach_mounts is not 1. 

It is probably easiest to let docker pull in runc package also during installation and that will make sure this knob is turned on.

Docker will depend on this feature so that we have less issues w.r.t mounts and device being busy.

Comment 12 Daniel Walsh 2017-07-07 10:01:13 UTC
Then we need to add 
Requires: runc

to docker package.

Comment 14 errata-xmlrpc 2017-08-02 00:17:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2350

Comment 15 Spyros Trigazis 2018-01-17 13:59:18 UTC
Fedora Atomic 26 and 27 are still affected, I'm not sure what need to be done.

I can't set may_detach_mounts to 1 in a fedora atomic 27 host with:
runc-1.0.0-12.rc4.gitaea4f21.fc27.x86_64

Should I open a bug for fedora?

Comment 16 Ed Santiago 2018-01-17 14:11:59 UTC
(In reply to Spyros Trigazis from comment #15)
> Fedora Atomic 26 and 27 are still affected

Can you clarify what you mean by "affected"? This issue affects only RHEL, and the /proc/sys/fs/may_detach_mounts switch exists only in RHEL. Fedora has never been affected by this issue and has never had a need for the may_detach_mounts option.

Comment 17 Spyros Trigazis 2018-01-17 14:49:56 UTC
If you:
* run kubernetes (tried with 1.7, 1.8 1.9 and 1.10.alpha) on fedora atomic 26 or 27 with docker as runtime
* have a pod with a secret or a configmap mounted
* try to delete a pod it gets stuck in state terminating

The issue is tracked here:
https://github.com/kubernetes/kubernetes/issues/51835

From the kubelet logs:
E0117 10:44:35.028204 6912 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/secret/e5fdeccf-fb72-11e7-9157-fa163eda350b-default-token-dtcng\" (\"e5fdeccf-fb72-11e7-9157-fa163eda350b\")" failed. No retries permitted until 2018-01-17 10:44:37.028168931 +0000 UTC m=+43.675199102 (durationBeforeRetry 2s). Error: "UnmountVolume.TearDown failed for volume \"default-token-dtcng\" (UniqueName: \"kubernetes.io/secret/e5fdeccf-fb72-11e7-9157-fa163eda350b-default-token-dtcng\") pod \"e5fdeccf-fb72-11e7-9157-fa163eda350b\" (UID: \"e5fdeccf-fb72-11e7-9157-fa163eda350b\") : remove /var/lib/kubelet/pods/e5fdeccf-fb72-11e7-9157-fa163eda350b/volumes/kubernetes.io~secret/default-token-dtcng: device or resource busy"

Comment 19 Antonio Murdaca 2018-10-09 15:28:42 UTC
This is not, somehow, working for rhel 7.5, Lokesh can you take a look? the related bug is: https://bugzilla.redhat.com/show_bug.cgi?id=1637623


Note You need to log in before you can comment on or make changes to this bug.