Bug 1336780 - [docker1.10] using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod"
Summary: [docker1.10] using network plugins \"redhat/openshift-ovs-multitenant\": Erro...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.2.1
Assignee: Scott Dodson
QA Contact: Ma xiaoqiang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-17 12:45 UTC by Marcel Wysocki
Modified: 2016-06-07 10:56 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Due to packaging changes in docker-1.9.1-40 containerized nodes did not have the correct docker components mounted from the host into the node container. This prevented pods from being correctly configured to use the SDN. The missing components have been added to the containerized node configuration.
Clone Of:
Environment:
Last Closed: 2016-06-07 10:56:23 UTC
Target Upstream Version:
Embargoed:
sdodson: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1208 0 normal SHIPPED_LIVE Red Hat OpenShift Enterprise atomic-openshift-utils bug fix update 2016-06-07 14:55:01 UTC

Description Marcel Wysocki 2016-05-17 12:45:11 UTC
Error syncing pod, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod"
122 times in the last 19 minutes
2

no pods can be run because of this.

Fresh install of OSE 3.2 on OpenStack.
All instances are in the default sec_group so nothing preventing them from communicating.

from oc describe:
  ---------	--------	-----	----						-------------	--------	------		-------
  18m		18m		1	{default-scheduler }						Normal		Scheduled	Successfully assigned router-2-deploy to infra-a2-mz0r8m82.test.osp.sfa.se
  16m		16m		2	{kubelet infra-a2-mz0r8m82.test.osp.sfa.se}			Warning		FailedSync	Error syncing pod, skipping: API error (500): Unknown device f509dec084d2b93b9d56a353a0af507694daa979387e3f2e57786fe7643ae314

  18m	13m	100	{kubelet infra-a2-mz0r8m82.test.osp.sfa.se}		Warning	FailedSync	Error syncing pod, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod"

  11m	11m	3	{kubelet infra-a2-mz0r8m82.test.osp.sfa.se}		Warning	FailedSync	Error syncing pod, skipping: API error (500): Unknown device 588d498f0657f22d4895d3c06375d461b1aa5d403299da026c787547918be051

  6m	6m	4	{kubelet infra-a2-mz0r8m82.test.osp.sfa.se}		Warning	FailedSync	Error syncing pod, skipping: API error (500): Unknown device 587d9df1dd0d567cc9aaab13918a8f3cc3d598b981dba80d089e414b44bb5901

  1m	1m	3	{kubelet infra-a2-mz0r8m82.test.osp.sfa.se}		Warning	FailedSync	Error syncing pod, skipping: API error (500): Unknown device 00e80638baee70435f669f56159f89173b5a56ac6015363665c58f29bcc00b3d

  12m	26s	151	{kubelet infra-a2-mz0r8m82.test.osp.sfa.se}		Warning	FailedSync	Error syncing pod, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod"



from the node's log:
May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:34.126083  107582 manager.go:1871] Failed to setup network for pod "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" using network plugins "redhat/openshift-ovs-multitenant": Error running network setup script: + network_mode=
May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: ; Skipping pod
May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:34.128064  107582 manager.go:1368] Killing container "d80e03a5042d753e9cd50f54a1645e7d386fabd085aa7473ef8453588d8fdf4b default/router-2-deploy" with 30 second grace period
May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:34.640278  107582 manager.go:1400] Container "d80e03a5042d753e9cd50f54a1645e7d386fabd085aa7473ef8453588d8fdf4b default/router-2-deploy" exited after 512.184744ms
May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:34.640345  107582 pod_workers.go:138] Error syncing pod cee383c2-1c29-11e6-9ea9-fa163e718c0a, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod"
May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:34.735081  107582 kubelet.go:2430] SyncLoop (PLEG): "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)", event: &pleg.PodLifecycleEvent{ID:"cee383c2-1c29-11e6-9ea9-fa163e718c0a", Type:"ContainerDied", Data:"d80e03a5042d753e9cd50f54a1645e7d386fabd085aa7473ef8453588d8fdf4b"}
May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:34.743332  107582 manager.go:1688] Need to restart pod infra container for "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" because it is not found
May 17 14:43:35 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:35.663163  107582 manager.go:1871] Failed to setup network for pod "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" using network plugins "redhat/openshift-ovs-multitenant": Error running network setup script: + network_mode=
May 17 14:43:35 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: ; Skipping pod
May 17 14:43:35 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:35.664496  107582 manager.go:1368] Killing container "474fac6e6f1c0e14c40e25dc2d015b0ce1ff4ad084bfb615180c5f2a93323c25 default/router-2-deploy" with 30 second grace period
May 17 14:43:35 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:35.990573  107582 manager.go:1400] Container "474fac6e6f1c0e14c40e25dc2d015b0ce1ff4ad084bfb615180c5f2a93323c25 default/router-2-deploy" exited after 326.046127ms
May 17 14:43:35 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:35.990639  107582 pod_workers.go:138] Error syncing pod cee383c2-1c29-11e6-9ea9-fa163e718c0a, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod"
May 17 14:43:36 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:36.057521  107582 kubelet.go:2430] SyncLoop (PLEG): "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)", event: &pleg.PodLifecycleEvent{ID:"cee383c2-1c29-11e6-9ea9-fa163e718c0a", Type:"ContainerDied", Data:"474fac6e6f1c0e14c40e25dc2d015b0ce1ff4ad084bfb615180c5f2a93323c25"}
May 17 14:43:36 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:36.064602  107582 manager.go:1688] Need to restart pod infra container for "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" because it is not found
May 17 14:43:36 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:36.993491  107582 manager.go:1871] Failed to setup network for pod "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" using network plugins "redhat/openshift-ovs-multitenant": Error running network setup script: + network_mode=
May 17 14:43:36 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: ; Skipping pod
May 17 14:43:36 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:36.994994  107582 manager.go:1368] Killing container "9f633b979992b9699415b791019667069266ac38cabf4c6cf5bcc6a53cbcf17a default/router-2-deploy" with 30 second grace period
May 17 14:43:37 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:37.365308  107582 manager.go:1400] Container "9f633b979992b9699415b791019667069266ac38cabf4c6cf5bcc6a53cbcf17a default/router-2-deploy" exited after 370.283939ms
May 17 14:43:37 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:37.365401  107582 pod_workers.go:138] Error syncing pod cee383c2-1c29-11e6-9ea9-fa163e718c0a, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod"
May 17 14:43:37 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:37.434341  107582 kubelet.go:2430] SyncLoop (PLEG): "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)", event: &pleg.PodLifecycleEvent{ID:"cee383c2-1c29-11e6-9ea9-fa163e718c0a", Type:"ContainerDied", Data:"9f633b979992b9699415b791019667069266ac38cabf4c6cf5bcc6a53cbcf17a"}
May 17 14:43:37 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:37.442229  107582 manager.go:1688] Need to restart pod infra container for "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" because it is not found
May 17 14:43:38 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:38.481398  107582 manager.go:1871] Failed to setup network for pod "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" using network plugins "redhat/openshift-ovs-multitenant": Error running network setup script: + network_mode=
May 17 14:43:51 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: ; Skipping pod
May 17 14:43:51 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:51.609003  107582 manager.go:1368] Killing container "f950205ab5366a6341e428bd0cd27a77da3a78c454bfd401a6faebf055d3241d default/router-2-deploy" with 30 second grace period
May 17 14:43:51 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:51.672250  107582 kubelet.go:2430] SyncLoop (PLEG): "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)", event: &pleg.PodLifecycleEvent{ID:"cee383c2-1c29-11e6-9ea9-fa163e718c0a", Type:"ContainerStarted", Data:"f950205ab5366a6341e428bd0cd27a77da3a78c454bfd401a6faebf055d3241d"}
May 17 14:43:51 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:51.990207  107582 manager.go:1400] Container "f950205ab5366a6341e428bd0cd27a77da3a78c454bfd401a6faebf055d3241d default/router-2-deploy" exited after 381.173568ms
May 17 14:43:51 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:51.990282  107582 pod_workers.go:138] Error syncing pod cee383c2-1c29-11e6-9ea9-fa163e718c0a, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod"
May 17 14:43:52 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:52.717289  107582 helpers.go:101] Unable to get network stats from pid 128752: couldn't read network stats: failure opening /rootfs/proc/128752/net/dev: open /rootfs/proc/128752/net/dev: no such file or directory
May 17 14:43:52 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:52.756698  107582 kubelet.go:2430] SyncLoop (PLEG): "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)", event: &pleg.PodLifecycleEvent{ID:"cee383c2-1c29-11e6-9ea9-fa163e718c0a", Type:"ContainerDied", Data:"f950205ab5366a6341e428bd0cd27a77da3a78c454bfd401a6faebf055d3241d"}
May 17 14:43:52 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:52.764190  107582 manager.go:1688] Need to restart pod infra container for "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" because it is not found
May 17 14:43:53 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:53.024043  107582 helpers.go:101] Unable to get network stats from pid 129194: couldn't read network stats: failure opening /rootfs/proc/129194/net/dev: open /rootfs/proc/129194/net/dev: no such file or directory
May 17 14:43:53 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:53.491400  107582 helpers.go:101] Unable to get network stats from pid 128127: couldn't read network stats: failure opening /rootfs/proc/128127/net/dev: open /rootfs/proc/128127/net/dev: no such file or directory
May 17 14:43:53 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:53.687749  107582 helpers.go:101] Unable to get network stats from pid 128275: couldn't read network stats: failure opening /rootfs/proc/128275/net/dev: open /rootfs/proc/128275/net/dev: no such file or directory
May 17 14:43:53 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:53.808700  107582 manager.go:1871] Failed to setup network for pod "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" using network plugins "redhat/openshift-ovs-multitenant": Error running network setup script: + network_mode=
May 17 14:43:53 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: ; Skipping pod
May 17 14:43:53 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:53.811988  107582 manager.go:1368] Killing container "01214ef91d00696cfd361a093528ed4eb1f3c5dcd582f51d767e77c5caff461b default/router-2-deploy" with 30 second grace period

Comment 1 Dan Williams 2016-05-17 14:53:11 UTC
network_mode=$(docker inspect --format "{{.HostConfig.NetworkMode}}" ${net_container})

which would imply that Docker reported a missing NetworkMode for the pod.

Can I get an 'oc describe pod xxxx' on that pod?

Can I also get a 'docker inspect XXX' on one of the router containers on the node, if openshift/kubernetes hasn't removed the container yet?  There might be a few around.

Comment 2 Marcel Wysocki 2016-05-17 16:00:13 UTC
-bash-4.2# docker ps -a | grep 929490c4416f
929490c4416f        openshift3/ose-pod:v3.2.0.20       "/pod"                   About a minute ago   Exited (0) About a minute ago                           k8s_POD.5667dc22_router-2-deploy_default_cee383c2-1c29-11e6-9ea9-fa163e718c0a_7c00b0d1
-bash-4.2# 

-bash-4.2# docker inspect --format "{{.HostConfig.NetworkMode}}" 929490c4416f
default
-bash-4.2# 

-bash-4.2# docker inspect --format "{{.HostConfig.NetworkMode}}" openvswitch
host
-bash-4.2# 
-bash-4.2# docker inspect --format "{{.HostConfig.NetworkMode}}" atomic-openshift-node
host
-bash-4.2# 

oc describe :

[cloud-user@installer-a1-63l7c77z ~]$ oc describe po/router-2-deploy |grep -v device
Name:		router-2-deploy
Namespace:	default
Node:		infra-a2-mz0r8m82.test.osp.sfa.se/192.168.192.117
Start Time:	Tue, 17 May 2016 14:20:57 +0200
Labels:		openshift.io/deployer-pod-for.name=router-2
Status:		Pending
IP:		
Controllers:	<none>
Containers:
  deployment:
    Container ID:	
    Image:		openshift3/ose-deployer:v3.2.0.20
    Image ID:		
    Port:		
    QoS Tier:
      cpu:		BestEffort
      memory:		BestEffort
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Environment Variables:
      KUBERNETES_MASTER:	https://master-a2-g9z9wzk9.test.osp.sfa.se:8443
      OPENSHIFT_MASTER:		https://master-a2-g9z9wzk9.test.osp.sfa.se:8443
      BEARER_TOKEN_FILE:	/var/run/secrets/kubernetes.io/serviceaccount/token
      OPENSHIFT_CA_DATA:	-----BEGIN CERTIFICATE-----
......
......
......
-----END CERTIFICATE-----

      OPENSHIFT_DEPLOYMENT_NAME:	router-2
      OPENSHIFT_DEPLOYMENT_NAMESPACE:	default
Conditions:
  Type		Status
  Ready 	False 
Volumes:
  deployer-token-snqxa:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	deployer-token-snqxa
Events:
  FirstSeen	LastSeen	Count	From						SubobjectPath	Type		Reason		Message
  ---------	--------	-----	----						-------------	--------	------		-------
  3h	51s	2511	{kubelet infra-a2-mz0r8m82.test.osp.sfa.se}		Warning	FailedSync	Error syncing pod, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod"
  2h	9s	44	{kubelet infra-a2-mz0r8m82.test.osp.sfa.se}		Warning	FailedSync	(events with common reason combined)


[cloud-user@installer-a1-63l7c77z ~]$ 

Docker inspect on one of the many failed container:

[
{
    "Id": "eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757",
    "Created": "2016-05-17T15:54:06.345692616Z",
    "Path": "/pod",
    "Args": [],
    "State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2016-05-17T15:54:07.20871088Z",
        "FinishedAt": "2016-05-17T15:54:07.253443045Z"
    },
    "Image": "2eda71099c5bb6a14f7e097b2369127cf0ec6689f08cef71a29e4ed4b426f505",
    "ResolvConfPath": "/var/lib/docker/containers/eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757/resolv.conf",
    "HostnamePath": "/var/lib/docker/containers/eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757/hostname",
    "HostsPath": "/var/lib/docker/containers/eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757/hosts",
    "LogPath": "/var/lib/docker/containers/eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757/eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757-json.log",
    "Name": "/k8s_POD.5667dc22_router-2-deploy_default_cee383c2-1c29-11e6-9ea9-fa163e718c0a_a896f8e5",
    "RestartCount": 0,
    "Driver": "devicemapper",
    "ExecDriver": "native-0.2",
    "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c1,c0",
    "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c1,c0",
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": null,
        "ContainerIDFile": "",
        "LxcConf": null,
        "Memory": 0,
        "MemoryReservation": 0,
        "MemorySwap": -1,
        "KernelMemory": 0,
        "CpuShares": 2,
        "CpuPeriod": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "CpuQuota": 0,
        "BlkioWeight": 0,
        "OomKillDisable": false,
        "MemorySwappiness": null,
        "Privileged": false,
        "PortBindings": null,
        "Links": null,
        "PublishAllPorts": false,
        "Dns": [
            "192.168.192.117",
            "192.168.192.117"
        ],
        "DnsOptions": null,
        "DnsSearch": [
            "default.svc.cluster.local",
            "svc.cluster.local",
            "cluster.local",
            "openstacklocal",
            "test.osp.sfa.se"
        ],
        "ExtraHosts": null,
        "VolumesFrom": null,
        "Devices": null,
        "NetworkMode": "default",
        "IpcMode": "",
        "PidMode": "",
        "UTSMode": "",
        "CapAdd": null,
        "CapDrop": null,
        "GroupAdd": null,
        "RestartPolicy": {
            "Name": "",
            "MaximumRetryCount": 0
        },
        "SecurityOpt": [
            "label:level:s0:c1,c0"
        ],
        "ReadonlyRootfs": false,
        "Ulimits": null,
        "Sysctls": null,
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "CgroupParent": "",
        "ConsoleSize": [
            0,
            0
        ],
        "VolumeDriver": "",
        "ShmSize": 67108864
    },
    "GraphDriver": {
        "Name": "devicemapper",
        "Data": {
            "DeviceId": "14123",
            "DeviceName": "docker-253:0-12601619-eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757",
            "DeviceSize": "107374182400"
        }
    },
    "Mounts": [],
    "Config": {
        "Hostname": "router-2-deploy",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "ROUTER_PORT_80_TCP_ADDR=172.30.58.254",
            "EAP_APP_PORT=tcp://172.30.125.143:8080",
            "KUBERNETES_PORT_53_UDP_PROTO=udp",
            "ROUTER_SERVICE_PORT_443_TCP=443",
            "EAP_APP_PORT_8080_TCP=tcp://172.30.125.143:8080",
            "KUBERNETES_PORT_443_TCP_PORT=443",
            "KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53",
            "ROUTER_SERVICE_HOST=172.30.58.254",
            "ROUTER_PORT_80_TCP_PROTO=tcp",
            "EAP_APP_PORT_8080_TCP_PORT=8080",
            "KUBERNETES_PORT_443_TCP_PROTO=tcp",
            "ROUTER_SERVICE_PORT_80_TCP=80",
            "ROUTER_PORT=tcp://172.30.58.254:80",
            "EAP_APP_SERVICE_PORT=8080",
            "ROUTER_PORT_443_TCP_PORT=443",
            "ROUTER_PORT_1936_TCP=tcp://172.30.58.254:1936",
            "ROUTER_PORT_1936_TCP_ADDR=172.30.58.254",
            "KUBERNETES_SERVICE_PORT=443",
            "ROUTER_SERVICE_PORT=80",
            "ROUTER_SERVICE_PORT_1936_TCP=1936",
            "ROUTER_PORT_80_TCP_PORT=80",
            "ROUTER_PORT_443_TCP=tcp://172.30.58.254:443",
            "KUBERNETES_SERVICE_PORT_DNS=53",
            "KUBERNETES_PORT_53_UDP=udp://172.30.0.1:53",
            "KUBERNETES_PORT_53_UDP_PORT=53",
            "ROUTER_PORT_1936_TCP_PORT=1936",
            "EAP_APP_PORT_8080_TCP_ADDR=172.30.125.143",
            "KUBERNETES_SERVICE_PORT_DNS_TCP=53",
            "KUBERNETES_PORT=tcp://172.30.0.1:443",
            "KUBERNETES_SERVICE_HOST=172.30.0.1",
            "KUBERNETES_SERVICE_PORT_HTTPS=443",
            "KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443",
            "KUBERNETES_PORT_53_TCP_PROTO=tcp",
            "ROUTER_PORT_443_TCP_PROTO=tcp",
            "ROUTER_PORT_443_TCP_ADDR=172.30.58.254",
            "EAP_APP_SERVICE_HOST=172.30.125.143",
            "EAP_APP_PORT_8080_TCP_PROTO=tcp",
            "KUBERNETES_PORT_53_TCP_PORT=53",
            "KUBERNETES_PORT_53_TCP_ADDR=172.30.0.1",
            "ROUTER_PORT_80_TCP=tcp://172.30.58.254:80",
            "ROUTER_PORT_1936_TCP_PROTO=tcp",
            "KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1",
            "KUBERNETES_PORT_53_UDP_ADDR=172.30.0.1",
            "container=docker",
            "PKGM=yum",
            "PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin"
        ],
        "Cmd": null,
        "Image": "openshift3/ose-pod:v3.2.0.20",
        "Volumes": null,
        "WorkingDir": "",
        "Entrypoint": [
            "/pod"
        ],
        "OnBuild": null,
        "Labels": {
            "Architecture": "x86_64",
            "Authoritative_Registry": "registry.access.redhat.com",
            "BZComponent": "openshift-enterprise-pod-docker",
            "Build_Host": "rcm-img-docker02.build.eng.bos.redhat.com",
            "Name": "openshift3/ose-pod",
            "Release": "3",
            "Vendor": "Red Hat, Inc.",
            "Version": "v3.2.0.20",
            "architecture": "x86_64",
            "build-date": "2016-05-05T13:52:32.008102Z",
            "distribution-scope": "public",
            "io.kubernetes.container.hash": "5667dc22",
            "io.kubernetes.container.name": "POD",
            "io.kubernetes.container.restartCount": "0",
            "io.kubernetes.container.terminationMessagePath": "",
            "io.kubernetes.pod.name": "router-2-deploy",
            "io.kubernetes.pod.namespace": "default",
            "io.kubernetes.pod.terminationGracePeriod": "30",
            "io.kubernetes.pod.uid": "cee383c2-1c29-11e6-9ea9-fa163e718c0a",
            "vcs-ref": "e10b66ab278fc5ad8c33961e025527d8c85e510a",
            "vcs-type": "git"
        }
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": null,
        "SandboxKey": "",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {
            "bridge": {
                "EndpointID": "",
                "Gateway": "",
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": ""
            }
        }
    }
}
]


by the way, this is running on RHEL-AH and containerized=true
also some output from the docker log when its starting the containers:

May 17 17:58:15 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:15.869868785+02:00" level=info msg="{Action=stop, ID=699f2ceb0cb4a05baf145b6994272d62068d9e58508f347798a0a7f6b2a1234b, LoginUID=4294967295, PID=107582}"
May 17 17:58:16 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:16.877941853+02:00" level=info msg="{Action=create, LoginUID=4294967295, PID=107582}"
May 17 17:58:17 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:17.656852036+02:00" level=info msg="{Action=start, ID=610b95ca7c7cce0cb43f6d41d48526235005fc61a9960c3f9f178db1f5852c4b, LoginUID=4294967295, PID=107582, Config={Hostname=router-2-deploy, AttachStdin=false, AttachStdout=false, AttachStderr=false, Tty=false, OpenStdin=false, StdinOnce=false, Env=[KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_53_TCP_PROTO=tcp ROUTER_PORT_443_TCP_PORT=443 KUBERNETES_SERVICE_PORT_DNS_TCP=53 KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443 ROUTER_PORT_80_TCP_PROTO=tcp EAP_APP_PORT_8080_TCP_PROTO=tcp KUBERNETES_SERVICE_HOST=172.30.0.1 KUBERNETES_PORT_53_TCP_PORT=53 ROUTER_SERVICE_HOST=172.30.58.254 ROUTER_SERVICE_PORT=80 ROUTER_SERVICE_PORT_1936_TCP=1936 ROUTER_PORT_80_TCP=tcp://172.30.58.254:80 ROUTER_PORT_1936_TCP_PORT=1936 ROUTER_PORT_1936_TCP_ADDR=172.30.58.254 KUBERNETES_PORT_53_UDP=udp://172.30.0.1:53 KUBERNETES_PORT_53_UDP_PROTO=udp ROUTER_SERVICE_PORT_80_TCP=80 ROUTER_PORT=tcp://172.30.58.254:80 ROUTER_PORT_1936_TCP_PROTO=tcp EAP_APP_SERVICE_PORT=8080 EAP_APP_PORT_8080_TCP_PORT=8080 KUBERNETES_PORT_53_UDP_ADDR=172.30.0.1 KUBERNETES_PORT_53_TCP_ADDR=172.30.0.1 KUBERNETES_PORT_53_UDP_PORT=53 KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53 ROUTER_PORT_443_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_PORT_HTTPS=443 EAP_APP_SERVICE_HOST=172.30.125.143 EAP_APP_PORT_8080_TCP=tcp://172.30.125.143:8080 ROUTER_PORT_443_TCP=tcp://172.30.58.254:443 ROUTER_PORT_443_TCP_ADDR=172.30.58.254 EAP_APP_PORT_8080_TCP_ADDR=172.30.125.143 ROUTER_PORT_80_TCP_PORT=80 EAP_APP_PORT=tcp://172.30.125.143:8080 KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1 ROUTER_SERVICE_PORT_443_TCP=443 ROUTER_PORT_80_TCP_ADDR=172.30.58.254 ROUTER_PORT_1936_TCP=tcp://172.30.58.254:1936 KUBERNETES_SERVICE_PORT_DNS=53 KUBERNETES_PORT=tcp://172.30.0.1:443 container=docker PKGM=yum PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin], Image=openshift3/ose-pod:v3.2.0.20, Entrypoint={parts:[/pod]}, NetworkDisabled=false, Labels=map[io.kubernetes.container.restartCount:0 Architecture:x86_64 architecture:x86_64 distribution-scope:public io.kubernetes.container.hash:5667dc22 build-date:2016-05-05T13:52:32.008102Z vcs-type:git io.kubernetes.pod.namespace:default Version:v3.2.0.20 io.kubernetes.container.terminationMessagePath: io.kubernetes.pod.name:router-2-deploy io.kubernetes.pod.uid:cee383c2-1c29-11e6-9ea9-fa163e718c0a Authoritative_Registry:registry.access.redhat.com Build_Host:rcm-img-docker02.build.eng.bos.redhat.com Release:3 vcs-ref:e10b66ab278fc5ad8c33961e025527d8c85e510a io.kubernetes.pod.terminationGracePeriod:30 io.kubernetes.container.name:POD BZComponent:openshift-enterprise-pod-docker Name:openshift3/ose-pod Vendor:Red Hat, Inc.]}, HostConfig={MemorySwap=-1, CPUShares=2, OomKillDisable=false, Privileged=false, PublishAllPorts=false, DNS=[192.168.192.117 192.168.192.117], DNSSearch=[default.svc.cluster.local svc.cluster.local cluster.local openstacklocal test.osp.sfa.se], NetworkMode=default, SecurityOpt=[label:level:s0:c1,c0], ReadonlyRootfs=false, LogConfig={Type:json-file Config:map[]}, ShmSize=67108864}}"
May 17 17:58:17 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: 2016/05/17 17:58:17 http: multiple response.WriteHeader calls
May 17 17:58:17 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:17.849488213+02:00" level=info msg="{Action=stop, ID=610b95ca7c7cce0cb43f6d41d48526235005fc61a9960c3f9f178db1f5852c4b, LoginUID=4294967295, PID=107582}"
May 17 17:58:19 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:19.019925361+02:00" level=info msg="{Action=create, LoginUID=4294967295, PID=107582}"
May 17 17:58:19 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:19.876768203+02:00" level=info msg="{Action=start, ID=a74cfcb04b2c581cfa0464f34ef8e1549e90270c7df5791742671bfad0cae84a, LoginUID=4294967295, PID=107582, Config={Hostname=router-2-deploy, AttachStdin=false, AttachStdout=false, AttachStderr=false, Tty=false, OpenStdin=false, StdinOnce=false, Env=[KUBERNETES_PORT_53_UDP_PORT=53 KUBERNETES_PORT_53_TCP_ADDR=172.30.0.1 ROUTER_PORT_80_TCP=tcp://172.30.58.254:80 EAP_APP_PORT=tcp://172.30.125.143:8080 KUBERNETES_PORT_53_TCP_PORT=53 KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53 ROUTER_SERVICE_PORT_80_TCP=80 ROUTER_SERVICE_PORT_443_TCP=443 ROUTER_PORT=tcp://172.30.58.254:80 ROUTER_SERVICE_PORT=80 ROUTER_SERVICE_PORT_1936_TCP=1936 EAP_APP_PORT_8080_TCP=tcp://172.30.125.143:8080 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_53_UDP=udp://172.30.0.1:53 KUBERNETES_PORT_53_UDP_ADDR=172.30.0.1 KUBERNETES_PORT_53_TCP_PROTO=tcp ROUTER_SERVICE_HOST=172.30.58.254 ROUTER_PORT_80_TCP_ADDR=172.30.58.254 ROUTER_PORT_443_TCP_PORT=443 ROUTER_PORT_1936_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_SERVICE_PORT_DNS=53 KUBERNETES_SERVICE_PORT_DNS_TCP=53 KUBERNETES_PORT=tcp://172.30.0.1:443 KUBERNETES_PORT_53_UDP_PROTO=udp ROUTER_PORT_80_TCP_PORT=80 EAP_APP_SERVICE_PORT=8080 EAP_APP_PORT_8080_TCP_PORT=8080 EAP_APP_PORT_8080_TCP_ADDR=172.30.125.143 KUBERNETES_SERVICE_HOST=172.30.0.1 ROUTER_PORT_80_TCP_PROTO=tcp ROUTER_PORT_443_TCP_ADDR=172.30.58.254 ROUTER_PORT_1936_TCP=tcp://172.30.58.254:1936 ROUTER_PORT_1936_TCP_ADDR=172.30.58.254 EAP_APP_SERVICE_HOST=172.30.125.143 KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1 ROUTER_PORT_443_TCP=tcp://172.30.58.254:443 ROUTER_PORT_443_TCP_PROTO=tcp EAP_APP_PORT_8080_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT=443 ROUTER_PORT_1936_TCP_PORT=1936 container=docker PKGM=yum PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin], Image=openshift3/ose-pod:v3.2.0.20, Entrypoint={parts:[/pod]}, NetworkDisabled=false, Labels=map[build-date:2016-05-05T13:52:32.008102Z vcs-type:git io.kubernetes.pod.namespace:default io.kubernetes.container.hash:5667dc22 Authoritative_Registry:registry.access.redhat.com Version:v3.2.0.20 io.kubernetes.pod.uid:cee383c2-1c29-11e6-9ea9-fa163e718c0a distribution-scope:public vcs-ref:e10b66ab278fc5ad8c33961e025527d8c85e510a io.kubernetes.pod.name:router-2-deploy io.kubernetes.pod.terminationGracePeriod:30 Release:3 Vendor:Red Hat, Inc. Build_Host:rcm-img-docker02.build.eng.bos.redhat.com Name:openshift3/ose-pod architecture:x86_64 io.kubernetes.container.terminationMessagePath: io.kubernetes.container.name:POD io.kubernetes.container.restartCount:0 Architecture:x86_64 BZComponent:openshift-enterprise-pod-docker]}, HostConfig={MemorySwap=-1, CPUShares=2, OomKillDisable=false, Privileged=false, PublishAllPorts=false, DNS=[192.168.192.117 192.168.192.117], DNSSearch=[default.svc.cluster.local svc.cluster.local cluster.local openstacklocal test.osp.sfa.se], NetworkMode=default, SecurityOpt=[label:level:s0:c1,c0], ReadonlyRootfs=false, LogConfig={Type:json-file Config:map[]}, ShmSize=67108864}}"
May 17 17:58:20 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: 2016/05/17 17:58:20 http: multiple response.WriteHeader calls
May 17 17:58:20 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:20.095609032+02:00" level=info msg="{Action=stop, ID=a74cfcb04b2c581cfa0464f34ef8e1549e90270c7df5791742671bfad0cae84a, LoginUID=4294967295, PID=107582}"
May 17 17:58:45 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:45.908218522+02:00" level=info msg="{Action=remove, ID=aabe1a1e3bf7b82caae86009ff77c3459cabfd97c0191d4bef24c56fa06ce107, LoginUID=4294967295, PID=107582}"

Comment 8 Marcel Wysocki 2016-05-18 11:30:25 UTC
Changing component since this is an installer issue

Comment 9 Scott Dodson 2016-05-18 13:34:42 UTC
(In reply to Marcel Wysocki from comment #7)
> https://github.com/openshift/openshift-ansible/pull/1913

You've confirmed this fixes it?

Comment 10 Marcel Wysocki 2016-05-18 15:05:14 UTC
yes. this approach fixes it and the other one mentioned in the PR.

Comment 11 Scott Dodson 2016-05-24 18:10:01 UTC
https://github.com/openshift/openshift-ansible/pull/1941 provides a cleaner fix that works both when /usr/bin/docker-current exists and when it doesn't.

Comment 13 Ma xiaoqiang 2016-05-25 06:58:52 UTC
Check on openshift-ansible-3.0.93-1

1. New library dependency is added to the docker-1.10 package, It is not enough to bind the docker and docker-current file to container.
# docker run -it -v /usr/bin/docker-current:/usr/bin/docker-current  --entrypoint bash registry.access.redhat.com/openshift3/node:v3.2 
[root@bc52ea75ba29 origin]# docker-current version
docker-current: error while loading shared libraries: libseccomp.so.2: cannot open shared object file: No such file or directory

2. If user use ose-3.1 on the latest Atomic, we bind the docker binary to container. This means that openshift run with the docker-1.9 or newer version.

Comment 14 Scott Dodson 2016-05-25 18:15:23 UTC
Dan,

Can the SDN use docker from /rootfs  whenever the env var OPENSHIFT_CONTAINERIZED == true rather than us trying to chase dependencies being mounted into the right places?

Comment 15 Scott Dodson 2016-05-25 20:53:50 UTC
Ma,

docker-1.10 is potentially not shipping but we'd definitely need to fix this if and when that happens. Are you ok with limiting the scope of this to only docker-1.9 which is currently supported maximum version? We should clone this.

Comment 16 Ma xiaoqiang 2016-05-26 06:55:11 UTC
Scott

How to deal with the second issue?  We still run ose-3.1.1.6 with docker-1.9 on Atomic.

Comment 18 Scott Dodson 2016-05-27 02:15:50 UTC
I've opened https://bugzilla.redhat.com/show_bug.cgi?id=1340324 to address the long term problem. Lets limit the scope of this bug to only fixing containerized installs on docker-1.9.1.

Comment 19 Ma xiaoqiang 2016-05-27 03:25:10 UTC
OK, move this issue to VERIFIED.

Comment 21 errata-xmlrpc 2016-06-07 10:56:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1208


Note You need to log in before you can comment on or make changes to this bug.