Bug 1441737

Summary: [runc spec] Enable kernel sysctl knob /proc/sys/fs/may_detach_mounts
Product: Red Hat Enterprise Linux 7 Reporter: Vivek Goyal <vgoyal>
Component: runcAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED ERRATA QA Contact: atomic-bugs <atomic-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.4CC: amurdaca, dwalsh, ebiederm, jhonce, lsm5, santiago, strigazi, vgoyal, viggiani
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: runc-1.0.0-12.1.gitf8ce01d.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1468249 (view as bug list) Environment:
Last Closed: 2017-08-02 00:17:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1441743, 1468249, 1542672    

Description Vivek Goyal 2017-04-12 15:11:55 UTC
Description of problem:

RHEL 7.4 kernel has introduced a new sysctl knob to control kernel behavior. This is called /proc/sys/fs/may_detach_mounts. This knob is set to value 0 by default. Container run times (docker and others) need the new behavior and
want it to be set to 1. 

So modify runc package to drop a file say /usr/lib/sysctl.d/99-docker.conf. Contents of this file can be say following.

fs.may_detach_mounts=1

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Vivek Goyal 2017-04-12 15:16:29 UTC
We need this change so that we can run docker daemon in host mount namespace. And that will enable shared volume feature of docker where volumes mounted by container can propagate to host (if user has configured it right).

Comment 4 Daniel Walsh 2017-04-12 16:14:04 UTC
What happens if this file gets placed on a system without the kernel mode?  I think it just gets ignored correct?  Ie we install the runc package on a RHEL7.3 OS>

Comment 5 Vivek Goyal 2017-04-12 17:26:19 UTC
I think that it will be ignored. IOW, I think systemd will try to write to this file but it will not be present. I am assuming that systemd will continue to write rest of the knobs.

I tried it on fedora kernel and I see following message in logs.

Apr 12 13:25:28 vm7-f25 systemd-sysctl[22835]: Couldn't write '1' to 'fs/may_detach_mounts', ignoring: No such file or directory

Comment 6 Daniel Walsh 2017-04-13 18:46:03 UTC
Lokesh lets drop  /usr/lib/sysctl.d/99-containers.conf  with this flag, for rhel7.4.

Comment 7 Ed Santiago 2017-06-08 15:18:48 UTC
This RPM install is not behaving as I had expected: the may_detach_mounts option is not taking effect until after a reboot. I believe the specfile is missing a %sysctl_apply directive:

    https://fedoraproject.org/wiki/Packaging:Guidelines#binfmt.d.2C_sysctl.d_and_tmpfiles.d

Comment 8 Vivek Goyal 2017-06-08 18:42:36 UTC
Good point Ed. 

Lokesh, we probably need to fix it.

Comment 11 Vivek Goyal 2017-07-06 13:02:39 UTC
This does not help with docker install. Docker package seems to ship its own runc and does not have dependency to install runc package. That means after installing docker, /proc/sys/fs/may_detach_mounts is not 1. 

It is probably easiest to let docker pull in runc package also during installation and that will make sure this knob is turned on.

Docker will depend on this feature so that we have less issues w.r.t mounts and device being busy.

Comment 12 Daniel Walsh 2017-07-07 10:01:13 UTC
Then we need to add 
Requires: runc

to docker package.

Comment 14 errata-xmlrpc 2017-08-02 00:17:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2350

Comment 15 Spyros Trigazis 2018-01-17 13:59:18 UTC
Fedora Atomic 26 and 27 are still affected, I'm not sure what need to be done.

I can't set may_detach_mounts to 1 in a fedora atomic 27 host with:
runc-1.0.0-12.rc4.gitaea4f21.fc27.x86_64

Should I open a bug for fedora?

Comment 16 Ed Santiago 2018-01-17 14:11:59 UTC
(In reply to Spyros Trigazis from comment #15)
> Fedora Atomic 26 and 27 are still affected

Can you clarify what you mean by "affected"? This issue affects only RHEL, and the /proc/sys/fs/may_detach_mounts switch exists only in RHEL. Fedora has never been affected by this issue and has never had a need for the may_detach_mounts option.

Comment 17 Spyros Trigazis 2018-01-17 14:49:56 UTC
If you:
* run kubernetes (tried with 1.7, 1.8 1.9 and 1.10.alpha) on fedora atomic 26 or 27 with docker as runtime
* have a pod with a secret or a configmap mounted
* try to delete a pod it gets stuck in state terminating

The issue is tracked here:
https://github.com/kubernetes/kubernetes/issues/51835

From the kubelet logs:
E0117 10:44:35.028204 6912 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/secret/e5fdeccf-fb72-11e7-9157-fa163eda350b-default-token-dtcng\" (\"e5fdeccf-fb72-11e7-9157-fa163eda350b\")" failed. No retries permitted until 2018-01-17 10:44:37.028168931 +0000 UTC m=+43.675199102 (durationBeforeRetry 2s). Error: "UnmountVolume.TearDown failed for volume \"default-token-dtcng\" (UniqueName: \"kubernetes.io/secret/e5fdeccf-fb72-11e7-9157-fa163eda350b-default-token-dtcng\") pod \"e5fdeccf-fb72-11e7-9157-fa163eda350b\" (UID: \"e5fdeccf-fb72-11e7-9157-fa163eda350b\") : remove /var/lib/kubelet/pods/e5fdeccf-fb72-11e7-9157-fa163eda350b/volumes/kubernetes.io~secret/default-token-dtcng: device or resource busy"

Comment 19 Antonio Murdaca 2018-10-09 15:28:42 UTC
This is not, somehow, working for rhel 7.5, Lokesh can you take a look? the related bug is: https://bugzilla.redhat.com/show_bug.cgi?id=1637623

Comment 20 Mimmus 2022-05-25 10:19:32 UTC
containerd.io package ships its own runc: does this package need to be fixed too?

Comment 21 Daniel Walsh 2022-05-25 10:24:39 UTC
Is that shipped by RHEL?