Bug 1375878

Summary:	[DOCKER] Cannot share net namespace of containers when enabling user ns remapping in docker 1.10
Product:	OpenShift Container Platform	Reporter:	Paul Weil <pweil>
Component:	Containers	Assignee:	Daniel Walsh <dwalsh>
Status:	CLOSED WONTFIX	QA Contact:	Chuan Yu <chuyu>
Severity:	low	Docs Contact:
Priority:	medium
Version:	unspecified	CC:	amurdaca, aos-bugs, dwalsh, imcleod, jokerman, jpazdziora, mmccomas, nagrawal
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-04-02 15:21:19 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Paul Weil 2016-09-14 07:16:38 UTC

Description of problem:

When enabling userns-remap=default in docker 1.10 you are unable to start containers in Kubernetes because you may not share the net namespace of the pause container.  You will receive an error from the daemon of 'Cannot share the host or a container's network namespace when user namespaces are enabled.'

This is fixed in 1.11 with https://github.com/docker/docker/pull/21383 so this bug is for consideration for backport if we intend on supporting userns-remap with 1.10.

Comment 1 Daniel Walsh 2016-10-14 18:05:48 UTC

Will need to be back ported if we can not go to docker-1.12 in the next release.

But for now I will close this as fixed in the next release.

Fixed in docker-1.12

Comment 3 Jan Pazdziora 2017-11-21 12:58:11 UTC

Could this be shifted to ON_QA for proper QE validation?

Comment 4 weiwei jiang 2017-11-22 08:02:57 UTC

Tried with 
# openshift version 
openshift v3.7.9
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8
# uname -a 
Linux host-172-16-120-75 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 (Maipo)
# docker info 
Containers: 253
 Running: 0
 Paused: 0
 Stopped: 253
Images: 3
Server Version: 1.12.6
Storage Driver: overlay2
 Backing Filesystem: xfs
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge overlay null host
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: runc docker-runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-693.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.4 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 3.702 GiB
Name: host-172-16-120-75
ID: WTPP:DTVB:6Z4C:5AAZ:AVHJ:NVNC:ZVQG:R3JH:6PTJ:VPZP:KU6V:U3Z3
Docker Root Dir: /var/lib/docker/165536.165536
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 29
 Goroutines: 48
 System Time: 2017-11-22T02:53:35.946693152-05:00
 EventsListeners: 0
Registry: https://registry.reg-aws.openshift.com:443/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 virt-openshift-05.lab.eng.nay.redhat.com:5000
 virt-openshift-05.lab.eng.nay.redhat.com:5001
 asb-registry.usersys.redhat.com:5000
 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888
 registry.reg-aws.openshift.com:443
 127.0.0.0/8
Registries: registry.reg-aws.openshift.com:443 (insecure), registry.access.redhat.com (secure), registry.access.redhat.com (secure), docker.io (secure)



And still can not make docker work well.

1. After append --userns-remap=default to the OPTION of /etc/sysconfig/docker, restart docker got issue https://github.com/moby/moby/issues/29659
2. After workaround with https://github.com/coreos/bugs/issues/1728, restart docker got error "Error starting daemon: error initializing graphdriver: Unable to take ownership of thin-pool (dockerVG-docker--pool) that already has used data blocks"
3. Then I switch to overlay2 to bypass the devicemapper, restart docker and got https://github.com/opencontainers/runc/issues/1130
4. Then I enable debug mode for docker, and found error message "nsenter: unable to unshare namespaces: Invalid argument" in the log

Comment 5 Antonio Murdaca 2018-03-09 17:18:12 UTC

Can QA test with latest docker/setup?

Comment 6 weiwei jiang 2018-03-19 09:14:02 UTC

Checked with
# openshift version
openshift v3.9.11
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.16
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
 selinux
 userns
Kernel Version: 3.10.0-860.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 1
Total Memory: 3.455 GiB
Name: qe-wjiang-master-container-etcd-1
ID: KXZX:DPVC:AKLU:W5QL:ADFZ:BUXQ:S5YI:O4C6:URFC:2ZCV:4R72:AJFF
Docker Root Dir: /var/lib/docker/165536.165536
Debug Mode (client): false
Debug Mode (server): false
Registry: https://registry.reg-aws.openshift.com:443/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
 virt-openshift-05.lab.eng.nay.redhat.com:5001
 asb-registry.usersys.redhat.com:5000
 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888
 registry.reg-aws.openshift.com:443
 virt-openshift-05.lab.eng.nay.redhat.com:5000
 127.0.0.0/8
Live Restore Enabled: false
Registries: registry.reg-aws.openshift.com:443 (insecure), registry.access.redhat.com (secure), registry.access.redhat.com (secure), docker.io (secure)


And still can not make docker work well
1. After append --userns-remap=default to the OPTION of /etc/sysconfig/docker, restart docker still got issue https://github.com/moby/moby/issues/29659
2. 2. After workaround with https://github.com/coreos/bugs/issues/1728, restart docker got error 

Mar 19 05:10:53 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:53.032553151-04:00" level=error msg="Handler for POST /v1.26/containers/create returned error: Cannot share the host's network namespace when user namespaces are enabled"
Mar 19 05:10:53 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:53.069662260-04:00" level=error msg="Handler for POST /v1.26/containers/create?name=atomic-openshift-master-api returned error: Privileged mode is incompatible with user names
paces. You must run the container in the host namespace (--userns=host) when running privileged mode."
Mar 19 05:10:53 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:53.071260210-04:00" level=error msg="Handler for POST /v1.26/containers/create returned error: Privileged mode is incompatible with user namespaces. You must run the container
 in the host namespace (--userns=host) when running privileged mode."
Mar 19 05:10:53 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:53.090651136-04:00" level=error msg="Handler for POST /v1.26/containers/etcd_container/stop returned error: No such container: etcd_container"
Mar 19 05:10:53 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:53.092206092-04:00" level=error msg="Handler for POST /v1.26/containers/etcd_container/stop returned error: No such container: etcd_container"
Mar 19 05:10:55 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:55.446941109-04:00" level=error msg="Handler for DELETE /v1.26/containers/openvswitch?force=1 returned error: No such container: openvswitch"
Mar 19 05:10:55 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:55.448586694-04:00" level=error msg="Handler for DELETE /v1.26/containers/openvswitch returned error: No such container: openvswitch"
Mar 19 05:10:55 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:55.484491405-04:00" level=error msg="Handler for POST /v1.26/containers/create?name=openvswitch returned error: Privileged mode is incompatible with user namespaces. You must 
run the container in the host namespace (--userns=host) when running privileged mode."
Mar 19 05:10:55 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:55.485662507-04:00" level=error msg="Handler for POST /v1.26/containers/create returned error: Privileged mode is incompatible with user namespaces. You must run the container
 in the host namespace (--userns=host) when running privileged mode."
Mar 19 05:10:57 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:57.379231820-04:00" level=error msg="Handler for POST /v1.26/containers/atomic-openshift-master-controllers/stop returned error: No such container: atomic-openshift-master-con
trollers"
Mar 19 05:10:57 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:57.379900582-04:00" level=error msg="Handler for POST /v1.26/containers/atomic-openshift-master-controllers/stop returned error: No such container: atomic-openshift-master-con
trollers"
Mar 19 05:10:58 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:58.188045787-04:00" level=error msg="Handler for DELETE /v1.26/containers/etcd_container?force=1 returned error: No such container: etcd_container"
Mar 19 05:10:58 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:58.189624718-04:00" level=error msg="Handler for DELETE /v1.26/containers/etcd_container returned error: No such container: etcd_container"
Mar 19 05:10:58 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:58.222641645-04:00" level=error msg="Handler for POST /v1.26/containers/create?name=etcd_container returned error: Cannot share the host's network namespace when user namespac
es are enabled"
Mar 19 05:10:58 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:58.224475742-04:00" level=error msg="Handler for POST /v1.26/containers/create returned error: Cannot share the host's network namespace when user namespaces are enabled"
Mar 19 05:10:58 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:58.255013328-04:00" level=error msg="Handler for POST /v1.26/containers/etcd_container/stop returned error: No such container: etcd_container"
Mar 19 05:10:58 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:10:58.256605431-04:00" level=error msg="Handler for POST /v1.26/containers/etcd_container/stop returned error: No such container: etcd_container"
Mar 19 05:11:00 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:00.332883901-04:00" level=error msg="Handler for POST /v1.26/containers/atomic-openshift-node/stop returned error: No such container: atomic-openshift-node"
Mar 19 05:11:00 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:00.333655624-04:00" level=error msg="Handler for POST /v1.26/containers/atomic-openshift-node/stop returned error: No such container: atomic-openshift-node"
Mar 19 05:11:00 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:00.497236484-04:00" level=error msg="Handler for POST /v1.26/containers/openvswitch/stop returned error: No such container: openvswitch"
Mar 19 05:11:00 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:00.498941551-04:00" level=error msg="Handler for POST /v1.26/containers/openvswitch/stop returned error: No such container: openvswitch"
Mar 19 05:11:03 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:03.049015478-04:00" level=error msg="Handler for POST /v1.26/containers/atomic-openshift-master-api/stop returned error: No such container: atomic-openshift-master-api"
Mar 19 05:11:03 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:03.050696905-04:00" level=error msg="Handler for POST /v1.26/containers/atomic-openshift-master-api/stop returned error: No such container: atomic-openshift-master-api"
Mar 19 05:11:03 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:03.084982113-04:00" level=error msg="Handler for DELETE /v1.26/containers/atomic-openshift-master-controllers?force=1 returned error: No such container: atomic-openshift-maste
r-controllers"
Mar 19 05:11:03 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:03.086647545-04:00" level=error msg="Handler for DELETE /v1.26/containers/atomic-openshift-master-controllers returned error: No such container: atomic-openshift-master-contro
llers"
Mar 19 05:11:03 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:03.122513591-04:00" level=error msg="Handler for POST /v1.26/containers/create?name=atomic-openshift-master-controllers returned error: Privileged mode is incompatible with us
er namespaces. You must run the container in the host namespace (--userns=host) when running privileged mode."
Mar 19 05:11:03 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:03.123856113-04:00" level=error msg="Handler for POST /v1.26/containers/create returned error: Privileged mode is incompatible with user namespaces. You must run the container
 in the host namespace (--userns=host) when running privileged mode."
Mar 19 05:11:03 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:03.438954374-04:00" level=error msg="Handler for DELETE /v1.26/containers/etcd_container?force=1 returned error: No such container: etcd_container"
Mar 19 05:11:03 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:03.439487121-04:00" level=error msg="Handler for DELETE /v1.26/containers/etcd_container returned error: No such container: etcd_container"
Mar 19 05:11:03 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:03.477596662-04:00" level=error msg="Handler for POST /v1.26/containers/create?name=etcd_container returned error: Cannot share the host's network namespace when user namespac
es are enabled"
Mar 19 05:11:03 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:03.479711673-04:00" level=error msg="Handler for POST /v1.26/containers/create returned error: Cannot share the host's network namespace when user namespaces are enabled"
Mar 19 05:11:03 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:03.512951489-04:00" level=error msg="Handler for POST /v1.26/containers/etcd_container/stop returned error: No such container: etcd_container"
Mar 19 05:11:03 qe-wjiang-master-container-etcd-1 dockerd-current[65611]: time="2018-03-19T05:11:03.514549544-04:00" level=error msg="Handler for POST /v1.26/containers/etcd_container/stop returned error: No such container: etcd_container"