Error syncing pod, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod" 122 times in the last 19 minutes 2 no pods can be run because of this. Fresh install of OSE 3.2 on OpenStack. All instances are in the default sec_group so nothing preventing them from communicating. from oc describe: --------- -------- ----- ---- ------------- -------- ------ ------- 18m 18m 1 {default-scheduler } Normal Scheduled Successfully assigned router-2-deploy to infra-a2-mz0r8m82.test.osp.sfa.se 16m 16m 2 {kubelet infra-a2-mz0r8m82.test.osp.sfa.se} Warning FailedSync Error syncing pod, skipping: API error (500): Unknown device f509dec084d2b93b9d56a353a0af507694daa979387e3f2e57786fe7643ae314 18m 13m 100 {kubelet infra-a2-mz0r8m82.test.osp.sfa.se} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod" 11m 11m 3 {kubelet infra-a2-mz0r8m82.test.osp.sfa.se} Warning FailedSync Error syncing pod, skipping: API error (500): Unknown device 588d498f0657f22d4895d3c06375d461b1aa5d403299da026c787547918be051 6m 6m 4 {kubelet infra-a2-mz0r8m82.test.osp.sfa.se} Warning FailedSync Error syncing pod, skipping: API error (500): Unknown device 587d9df1dd0d567cc9aaab13918a8f3cc3d598b981dba80d089e414b44bb5901 1m 1m 3 {kubelet infra-a2-mz0r8m82.test.osp.sfa.se} Warning FailedSync Error syncing pod, skipping: API error (500): Unknown device 00e80638baee70435f669f56159f89173b5a56ac6015363665c58f29bcc00b3d 12m 26s 151 {kubelet infra-a2-mz0r8m82.test.osp.sfa.se} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod" from the node's log: May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:34.126083 107582 manager.go:1871] Failed to setup network for pod "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" using network plugins "redhat/openshift-ovs-multitenant": Error running network setup script: + network_mode= May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: ; Skipping pod May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:34.128064 107582 manager.go:1368] Killing container "d80e03a5042d753e9cd50f54a1645e7d386fabd085aa7473ef8453588d8fdf4b default/router-2-deploy" with 30 second grace period May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:34.640278 107582 manager.go:1400] Container "d80e03a5042d753e9cd50f54a1645e7d386fabd085aa7473ef8453588d8fdf4b default/router-2-deploy" exited after 512.184744ms May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:34.640345 107582 pod_workers.go:138] Error syncing pod cee383c2-1c29-11e6-9ea9-fa163e718c0a, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod" May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:34.735081 107582 kubelet.go:2430] SyncLoop (PLEG): "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)", event: &pleg.PodLifecycleEvent{ID:"cee383c2-1c29-11e6-9ea9-fa163e718c0a", Type:"ContainerDied", Data:"d80e03a5042d753e9cd50f54a1645e7d386fabd085aa7473ef8453588d8fdf4b"} May 17 14:43:34 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:34.743332 107582 manager.go:1688] Need to restart pod infra container for "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" because it is not found May 17 14:43:35 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:35.663163 107582 manager.go:1871] Failed to setup network for pod "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" using network plugins "redhat/openshift-ovs-multitenant": Error running network setup script: + network_mode= May 17 14:43:35 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: ; Skipping pod May 17 14:43:35 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:35.664496 107582 manager.go:1368] Killing container "474fac6e6f1c0e14c40e25dc2d015b0ce1ff4ad084bfb615180c5f2a93323c25 default/router-2-deploy" with 30 second grace period May 17 14:43:35 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:35.990573 107582 manager.go:1400] Container "474fac6e6f1c0e14c40e25dc2d015b0ce1ff4ad084bfb615180c5f2a93323c25 default/router-2-deploy" exited after 326.046127ms May 17 14:43:35 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:35.990639 107582 pod_workers.go:138] Error syncing pod cee383c2-1c29-11e6-9ea9-fa163e718c0a, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod" May 17 14:43:36 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:36.057521 107582 kubelet.go:2430] SyncLoop (PLEG): "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)", event: &pleg.PodLifecycleEvent{ID:"cee383c2-1c29-11e6-9ea9-fa163e718c0a", Type:"ContainerDied", Data:"474fac6e6f1c0e14c40e25dc2d015b0ce1ff4ad084bfb615180c5f2a93323c25"} May 17 14:43:36 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:36.064602 107582 manager.go:1688] Need to restart pod infra container for "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" because it is not found May 17 14:43:36 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:36.993491 107582 manager.go:1871] Failed to setup network for pod "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" using network plugins "redhat/openshift-ovs-multitenant": Error running network setup script: + network_mode= May 17 14:43:36 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: ; Skipping pod May 17 14:43:36 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:36.994994 107582 manager.go:1368] Killing container "9f633b979992b9699415b791019667069266ac38cabf4c6cf5bcc6a53cbcf17a default/router-2-deploy" with 30 second grace period May 17 14:43:37 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:37.365308 107582 manager.go:1400] Container "9f633b979992b9699415b791019667069266ac38cabf4c6cf5bcc6a53cbcf17a default/router-2-deploy" exited after 370.283939ms May 17 14:43:37 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:37.365401 107582 pod_workers.go:138] Error syncing pod cee383c2-1c29-11e6-9ea9-fa163e718c0a, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod" May 17 14:43:37 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:37.434341 107582 kubelet.go:2430] SyncLoop (PLEG): "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)", event: &pleg.PodLifecycleEvent{ID:"cee383c2-1c29-11e6-9ea9-fa163e718c0a", Type:"ContainerDied", Data:"9f633b979992b9699415b791019667069266ac38cabf4c6cf5bcc6a53cbcf17a"} May 17 14:43:37 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:37.442229 107582 manager.go:1688] Need to restart pod infra container for "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" because it is not found May 17 14:43:38 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:38.481398 107582 manager.go:1871] Failed to setup network for pod "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" using network plugins "redhat/openshift-ovs-multitenant": Error running network setup script: + network_mode= May 17 14:43:51 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: ; Skipping pod May 17 14:43:51 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:51.609003 107582 manager.go:1368] Killing container "f950205ab5366a6341e428bd0cd27a77da3a78c454bfd401a6faebf055d3241d default/router-2-deploy" with 30 second grace period May 17 14:43:51 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:51.672250 107582 kubelet.go:2430] SyncLoop (PLEG): "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)", event: &pleg.PodLifecycleEvent{ID:"cee383c2-1c29-11e6-9ea9-fa163e718c0a", Type:"ContainerStarted", Data:"f950205ab5366a6341e428bd0cd27a77da3a78c454bfd401a6faebf055d3241d"} May 17 14:43:51 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:51.990207 107582 manager.go:1400] Container "f950205ab5366a6341e428bd0cd27a77da3a78c454bfd401a6faebf055d3241d default/router-2-deploy" exited after 381.173568ms May 17 14:43:51 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:51.990282 107582 pod_workers.go:138] Error syncing pod cee383c2-1c29-11e6-9ea9-fa163e718c0a, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod" May 17 14:43:52 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:52.717289 107582 helpers.go:101] Unable to get network stats from pid 128752: couldn't read network stats: failure opening /rootfs/proc/128752/net/dev: open /rootfs/proc/128752/net/dev: no such file or directory May 17 14:43:52 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:52.756698 107582 kubelet.go:2430] SyncLoop (PLEG): "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)", event: &pleg.PodLifecycleEvent{ID:"cee383c2-1c29-11e6-9ea9-fa163e718c0a", Type:"ContainerDied", Data:"f950205ab5366a6341e428bd0cd27a77da3a78c454bfd401a6faebf055d3241d"} May 17 14:43:52 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:52.764190 107582 manager.go:1688] Need to restart pod infra container for "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" because it is not found May 17 14:43:53 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:53.024043 107582 helpers.go:101] Unable to get network stats from pid 129194: couldn't read network stats: failure opening /rootfs/proc/129194/net/dev: open /rootfs/proc/129194/net/dev: no such file or directory May 17 14:43:53 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:53.491400 107582 helpers.go:101] Unable to get network stats from pid 128127: couldn't read network stats: failure opening /rootfs/proc/128127/net/dev: open /rootfs/proc/128127/net/dev: no such file or directory May 17 14:43:53 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:53.687749 107582 helpers.go:101] Unable to get network stats from pid 128275: couldn't read network stats: failure opening /rootfs/proc/128275/net/dev: open /rootfs/proc/128275/net/dev: no such file or directory May 17 14:43:53 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: E0517 14:43:53.808700 107582 manager.go:1871] Failed to setup network for pod "router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)" using network plugins "redhat/openshift-ovs-multitenant": Error running network setup script: + network_mode= May 17 14:43:53 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: ; Skipping pod May 17 14:43:53 infra-a2-mz0r8m82.test.osp.sfa.se atomic-openshift-node[107538]: I0517 14:43:53.811988 107582 manager.go:1368] Killing container "01214ef91d00696cfd361a093528ed4eb1f3c5dcd582f51d767e77c5caff461b default/router-2-deploy" with 30 second grace period
network_mode=$(docker inspect --format "{{.HostConfig.NetworkMode}}" ${net_container}) which would imply that Docker reported a missing NetworkMode for the pod. Can I get an 'oc describe pod xxxx' on that pod? Can I also get a 'docker inspect XXX' on one of the router containers on the node, if openshift/kubernetes hasn't removed the container yet? There might be a few around.
-bash-4.2# docker ps -a | grep 929490c4416f 929490c4416f openshift3/ose-pod:v3.2.0.20 "/pod" About a minute ago Exited (0) About a minute ago k8s_POD.5667dc22_router-2-deploy_default_cee383c2-1c29-11e6-9ea9-fa163e718c0a_7c00b0d1 -bash-4.2# -bash-4.2# docker inspect --format "{{.HostConfig.NetworkMode}}" 929490c4416f default -bash-4.2# -bash-4.2# docker inspect --format "{{.HostConfig.NetworkMode}}" openvswitch host -bash-4.2# -bash-4.2# docker inspect --format "{{.HostConfig.NetworkMode}}" atomic-openshift-node host -bash-4.2# oc describe : [cloud-user@installer-a1-63l7c77z ~]$ oc describe po/router-2-deploy |grep -v device Name: router-2-deploy Namespace: default Node: infra-a2-mz0r8m82.test.osp.sfa.se/192.168.192.117 Start Time: Tue, 17 May 2016 14:20:57 +0200 Labels: openshift.io/deployer-pod-for.name=router-2 Status: Pending IP: Controllers: <none> Containers: deployment: Container ID: Image: openshift3/ose-deployer:v3.2.0.20 Image ID: Port: QoS Tier: cpu: BestEffort memory: BestEffort State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: KUBERNETES_MASTER: https://master-a2-g9z9wzk9.test.osp.sfa.se:8443 OPENSHIFT_MASTER: https://master-a2-g9z9wzk9.test.osp.sfa.se:8443 BEARER_TOKEN_FILE: /var/run/secrets/kubernetes.io/serviceaccount/token OPENSHIFT_CA_DATA: -----BEGIN CERTIFICATE----- ...... ...... ...... -----END CERTIFICATE----- OPENSHIFT_DEPLOYMENT_NAME: router-2 OPENSHIFT_DEPLOYMENT_NAMESPACE: default Conditions: Type Status Ready False Volumes: deployer-token-snqxa: Type: Secret (a volume populated by a Secret) SecretName: deployer-token-snqxa Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3h 51s 2511 {kubelet infra-a2-mz0r8m82.test.osp.sfa.se} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "router-2-deploy_default" with SetupNetworkError: "Failed to setup network for pod \"router-2-deploy_default(cee383c2-1c29-11e6-9ea9-fa163e718c0a)\" using network plugins \"redhat/openshift-ovs-multitenant\": Error running network setup script: + network_mode=\n; Skipping pod" 2h 9s 44 {kubelet infra-a2-mz0r8m82.test.osp.sfa.se} Warning FailedSync (events with common reason combined) [cloud-user@installer-a1-63l7c77z ~]$ Docker inspect on one of the many failed container: [ { "Id": "eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757", "Created": "2016-05-17T15:54:06.345692616Z", "Path": "/pod", "Args": [], "State": { "Status": "exited", "Running": false, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 0, "ExitCode": 0, "Error": "", "StartedAt": "2016-05-17T15:54:07.20871088Z", "FinishedAt": "2016-05-17T15:54:07.253443045Z" }, "Image": "2eda71099c5bb6a14f7e097b2369127cf0ec6689f08cef71a29e4ed4b426f505", "ResolvConfPath": "/var/lib/docker/containers/eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757/resolv.conf", "HostnamePath": "/var/lib/docker/containers/eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757/hostname", "HostsPath": "/var/lib/docker/containers/eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757/hosts", "LogPath": "/var/lib/docker/containers/eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757/eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757-json.log", "Name": "/k8s_POD.5667dc22_router-2-deploy_default_cee383c2-1c29-11e6-9ea9-fa163e718c0a_a896f8e5", "RestartCount": 0, "Driver": "devicemapper", "ExecDriver": "native-0.2", "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c1,c0", "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c1,c0", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": null, "ContainerIDFile": "", "LxcConf": null, "Memory": 0, "MemoryReservation": 0, "MemorySwap": -1, "KernelMemory": 0, "CpuShares": 2, "CpuPeriod": 0, "CpusetCpus": "", "CpusetMems": "", "CpuQuota": 0, "BlkioWeight": 0, "OomKillDisable": false, "MemorySwappiness": null, "Privileged": false, "PortBindings": null, "Links": null, "PublishAllPorts": false, "Dns": [ "192.168.192.117", "192.168.192.117" ], "DnsOptions": null, "DnsSearch": [ "default.svc.cluster.local", "svc.cluster.local", "cluster.local", "openstacklocal", "test.osp.sfa.se" ], "ExtraHosts": null, "VolumesFrom": null, "Devices": null, "NetworkMode": "default", "IpcMode": "", "PidMode": "", "UTSMode": "", "CapAdd": null, "CapDrop": null, "GroupAdd": null, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "SecurityOpt": [ "label:level:s0:c1,c0" ], "ReadonlyRootfs": false, "Ulimits": null, "Sysctls": null, "LogConfig": { "Type": "json-file", "Config": {} }, "CgroupParent": "", "ConsoleSize": [ 0, 0 ], "VolumeDriver": "", "ShmSize": 67108864 }, "GraphDriver": { "Name": "devicemapper", "Data": { "DeviceId": "14123", "DeviceName": "docker-253:0-12601619-eac0422bb8b692748c7186bfd78aee4b1f6ab0d6e3dbac4b8870bb0a45f13757", "DeviceSize": "107374182400" } }, "Mounts": [], "Config": { "Hostname": "router-2-deploy", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "ROUTER_PORT_80_TCP_ADDR=172.30.58.254", "EAP_APP_PORT=tcp://172.30.125.143:8080", "KUBERNETES_PORT_53_UDP_PROTO=udp", "ROUTER_SERVICE_PORT_443_TCP=443", "EAP_APP_PORT_8080_TCP=tcp://172.30.125.143:8080", "KUBERNETES_PORT_443_TCP_PORT=443", "KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53", "ROUTER_SERVICE_HOST=172.30.58.254", "ROUTER_PORT_80_TCP_PROTO=tcp", "EAP_APP_PORT_8080_TCP_PORT=8080", "KUBERNETES_PORT_443_TCP_PROTO=tcp", "ROUTER_SERVICE_PORT_80_TCP=80", "ROUTER_PORT=tcp://172.30.58.254:80", "EAP_APP_SERVICE_PORT=8080", "ROUTER_PORT_443_TCP_PORT=443", "ROUTER_PORT_1936_TCP=tcp://172.30.58.254:1936", "ROUTER_PORT_1936_TCP_ADDR=172.30.58.254", "KUBERNETES_SERVICE_PORT=443", "ROUTER_SERVICE_PORT=80", "ROUTER_SERVICE_PORT_1936_TCP=1936", "ROUTER_PORT_80_TCP_PORT=80", "ROUTER_PORT_443_TCP=tcp://172.30.58.254:443", "KUBERNETES_SERVICE_PORT_DNS=53", "KUBERNETES_PORT_53_UDP=udp://172.30.0.1:53", "KUBERNETES_PORT_53_UDP_PORT=53", "ROUTER_PORT_1936_TCP_PORT=1936", "EAP_APP_PORT_8080_TCP_ADDR=172.30.125.143", "KUBERNETES_SERVICE_PORT_DNS_TCP=53", "KUBERNETES_PORT=tcp://172.30.0.1:443", "KUBERNETES_SERVICE_HOST=172.30.0.1", "KUBERNETES_SERVICE_PORT_HTTPS=443", "KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443", "KUBERNETES_PORT_53_TCP_PROTO=tcp", "ROUTER_PORT_443_TCP_PROTO=tcp", "ROUTER_PORT_443_TCP_ADDR=172.30.58.254", "EAP_APP_SERVICE_HOST=172.30.125.143", "EAP_APP_PORT_8080_TCP_PROTO=tcp", "KUBERNETES_PORT_53_TCP_PORT=53", "KUBERNETES_PORT_53_TCP_ADDR=172.30.0.1", "ROUTER_PORT_80_TCP=tcp://172.30.58.254:80", "ROUTER_PORT_1936_TCP_PROTO=tcp", "KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1", "KUBERNETES_PORT_53_UDP_ADDR=172.30.0.1", "container=docker", "PKGM=yum", "PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin" ], "Cmd": null, "Image": "openshift3/ose-pod:v3.2.0.20", "Volumes": null, "WorkingDir": "", "Entrypoint": [ "/pod" ], "OnBuild": null, "Labels": { "Architecture": "x86_64", "Authoritative_Registry": "registry.access.redhat.com", "BZComponent": "openshift-enterprise-pod-docker", "Build_Host": "rcm-img-docker02.build.eng.bos.redhat.com", "Name": "openshift3/ose-pod", "Release": "3", "Vendor": "Red Hat, Inc.", "Version": "v3.2.0.20", "architecture": "x86_64", "build-date": "2016-05-05T13:52:32.008102Z", "distribution-scope": "public", "io.kubernetes.container.hash": "5667dc22", "io.kubernetes.container.name": "POD", "io.kubernetes.container.restartCount": "0", "io.kubernetes.container.terminationMessagePath": "", "io.kubernetes.pod.name": "router-2-deploy", "io.kubernetes.pod.namespace": "default", "io.kubernetes.pod.terminationGracePeriod": "30", "io.kubernetes.pod.uid": "cee383c2-1c29-11e6-9ea9-fa163e718c0a", "vcs-ref": "e10b66ab278fc5ad8c33961e025527d8c85e510a", "vcs-type": "git" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": null, "SandboxKey": "", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "bridge": { "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "" } } } } ] by the way, this is running on RHEL-AH and containerized=true also some output from the docker log when its starting the containers: May 17 17:58:15 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:15.869868785+02:00" level=info msg="{Action=stop, ID=699f2ceb0cb4a05baf145b6994272d62068d9e58508f347798a0a7f6b2a1234b, LoginUID=4294967295, PID=107582}" May 17 17:58:16 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:16.877941853+02:00" level=info msg="{Action=create, LoginUID=4294967295, PID=107582}" May 17 17:58:17 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:17.656852036+02:00" level=info msg="{Action=start, ID=610b95ca7c7cce0cb43f6d41d48526235005fc61a9960c3f9f178db1f5852c4b, LoginUID=4294967295, PID=107582, Config={Hostname=router-2-deploy, AttachStdin=false, AttachStdout=false, AttachStderr=false, Tty=false, OpenStdin=false, StdinOnce=false, Env=[KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_53_TCP_PROTO=tcp ROUTER_PORT_443_TCP_PORT=443 KUBERNETES_SERVICE_PORT_DNS_TCP=53 KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443 ROUTER_PORT_80_TCP_PROTO=tcp EAP_APP_PORT_8080_TCP_PROTO=tcp KUBERNETES_SERVICE_HOST=172.30.0.1 KUBERNETES_PORT_53_TCP_PORT=53 ROUTER_SERVICE_HOST=172.30.58.254 ROUTER_SERVICE_PORT=80 ROUTER_SERVICE_PORT_1936_TCP=1936 ROUTER_PORT_80_TCP=tcp://172.30.58.254:80 ROUTER_PORT_1936_TCP_PORT=1936 ROUTER_PORT_1936_TCP_ADDR=172.30.58.254 KUBERNETES_PORT_53_UDP=udp://172.30.0.1:53 KUBERNETES_PORT_53_UDP_PROTO=udp ROUTER_SERVICE_PORT_80_TCP=80 ROUTER_PORT=tcp://172.30.58.254:80 ROUTER_PORT_1936_TCP_PROTO=tcp EAP_APP_SERVICE_PORT=8080 EAP_APP_PORT_8080_TCP_PORT=8080 KUBERNETES_PORT_53_UDP_ADDR=172.30.0.1 KUBERNETES_PORT_53_TCP_ADDR=172.30.0.1 KUBERNETES_PORT_53_UDP_PORT=53 KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53 ROUTER_PORT_443_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_PORT_HTTPS=443 EAP_APP_SERVICE_HOST=172.30.125.143 EAP_APP_PORT_8080_TCP=tcp://172.30.125.143:8080 ROUTER_PORT_443_TCP=tcp://172.30.58.254:443 ROUTER_PORT_443_TCP_ADDR=172.30.58.254 EAP_APP_PORT_8080_TCP_ADDR=172.30.125.143 ROUTER_PORT_80_TCP_PORT=80 EAP_APP_PORT=tcp://172.30.125.143:8080 KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1 ROUTER_SERVICE_PORT_443_TCP=443 ROUTER_PORT_80_TCP_ADDR=172.30.58.254 ROUTER_PORT_1936_TCP=tcp://172.30.58.254:1936 KUBERNETES_SERVICE_PORT_DNS=53 KUBERNETES_PORT=tcp://172.30.0.1:443 container=docker PKGM=yum PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin], Image=openshift3/ose-pod:v3.2.0.20, Entrypoint={parts:[/pod]}, NetworkDisabled=false, Labels=map[io.kubernetes.container.restartCount:0 Architecture:x86_64 architecture:x86_64 distribution-scope:public io.kubernetes.container.hash:5667dc22 build-date:2016-05-05T13:52:32.008102Z vcs-type:git io.kubernetes.pod.namespace:default Version:v3.2.0.20 io.kubernetes.container.terminationMessagePath: io.kubernetes.pod.name:router-2-deploy io.kubernetes.pod.uid:cee383c2-1c29-11e6-9ea9-fa163e718c0a Authoritative_Registry:registry.access.redhat.com Build_Host:rcm-img-docker02.build.eng.bos.redhat.com Release:3 vcs-ref:e10b66ab278fc5ad8c33961e025527d8c85e510a io.kubernetes.pod.terminationGracePeriod:30 io.kubernetes.container.name:POD BZComponent:openshift-enterprise-pod-docker Name:openshift3/ose-pod Vendor:Red Hat, Inc.]}, HostConfig={MemorySwap=-1, CPUShares=2, OomKillDisable=false, Privileged=false, PublishAllPorts=false, DNS=[192.168.192.117 192.168.192.117], DNSSearch=[default.svc.cluster.local svc.cluster.local cluster.local openstacklocal test.osp.sfa.se], NetworkMode=default, SecurityOpt=[label:level:s0:c1,c0], ReadonlyRootfs=false, LogConfig={Type:json-file Config:map[]}, ShmSize=67108864}}" May 17 17:58:17 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: 2016/05/17 17:58:17 http: multiple response.WriteHeader calls May 17 17:58:17 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:17.849488213+02:00" level=info msg="{Action=stop, ID=610b95ca7c7cce0cb43f6d41d48526235005fc61a9960c3f9f178db1f5852c4b, LoginUID=4294967295, PID=107582}" May 17 17:58:19 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:19.019925361+02:00" level=info msg="{Action=create, LoginUID=4294967295, PID=107582}" May 17 17:58:19 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:19.876768203+02:00" level=info msg="{Action=start, ID=a74cfcb04b2c581cfa0464f34ef8e1549e90270c7df5791742671bfad0cae84a, LoginUID=4294967295, PID=107582, Config={Hostname=router-2-deploy, AttachStdin=false, AttachStdout=false, AttachStderr=false, Tty=false, OpenStdin=false, StdinOnce=false, Env=[KUBERNETES_PORT_53_UDP_PORT=53 KUBERNETES_PORT_53_TCP_ADDR=172.30.0.1 ROUTER_PORT_80_TCP=tcp://172.30.58.254:80 EAP_APP_PORT=tcp://172.30.125.143:8080 KUBERNETES_PORT_53_TCP_PORT=53 KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53 ROUTER_SERVICE_PORT_80_TCP=80 ROUTER_SERVICE_PORT_443_TCP=443 ROUTER_PORT=tcp://172.30.58.254:80 ROUTER_SERVICE_PORT=80 ROUTER_SERVICE_PORT_1936_TCP=1936 EAP_APP_PORT_8080_TCP=tcp://172.30.125.143:8080 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_53_UDP=udp://172.30.0.1:53 KUBERNETES_PORT_53_UDP_ADDR=172.30.0.1 KUBERNETES_PORT_53_TCP_PROTO=tcp ROUTER_SERVICE_HOST=172.30.58.254 ROUTER_PORT_80_TCP_ADDR=172.30.58.254 ROUTER_PORT_443_TCP_PORT=443 ROUTER_PORT_1936_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_SERVICE_PORT_DNS=53 KUBERNETES_SERVICE_PORT_DNS_TCP=53 KUBERNETES_PORT=tcp://172.30.0.1:443 KUBERNETES_PORT_53_UDP_PROTO=udp ROUTER_PORT_80_TCP_PORT=80 EAP_APP_SERVICE_PORT=8080 EAP_APP_PORT_8080_TCP_PORT=8080 EAP_APP_PORT_8080_TCP_ADDR=172.30.125.143 KUBERNETES_SERVICE_HOST=172.30.0.1 ROUTER_PORT_80_TCP_PROTO=tcp ROUTER_PORT_443_TCP_ADDR=172.30.58.254 ROUTER_PORT_1936_TCP=tcp://172.30.58.254:1936 ROUTER_PORT_1936_TCP_ADDR=172.30.58.254 EAP_APP_SERVICE_HOST=172.30.125.143 KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1 ROUTER_PORT_443_TCP=tcp://172.30.58.254:443 ROUTER_PORT_443_TCP_PROTO=tcp EAP_APP_PORT_8080_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT=443 ROUTER_PORT_1936_TCP_PORT=1936 container=docker PKGM=yum PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin], Image=openshift3/ose-pod:v3.2.0.20, Entrypoint={parts:[/pod]}, NetworkDisabled=false, Labels=map[build-date:2016-05-05T13:52:32.008102Z vcs-type:git io.kubernetes.pod.namespace:default io.kubernetes.container.hash:5667dc22 Authoritative_Registry:registry.access.redhat.com Version:v3.2.0.20 io.kubernetes.pod.uid:cee383c2-1c29-11e6-9ea9-fa163e718c0a distribution-scope:public vcs-ref:e10b66ab278fc5ad8c33961e025527d8c85e510a io.kubernetes.pod.name:router-2-deploy io.kubernetes.pod.terminationGracePeriod:30 Release:3 Vendor:Red Hat, Inc. Build_Host:rcm-img-docker02.build.eng.bos.redhat.com Name:openshift3/ose-pod architecture:x86_64 io.kubernetes.container.terminationMessagePath: io.kubernetes.container.name:POD io.kubernetes.container.restartCount:0 Architecture:x86_64 BZComponent:openshift-enterprise-pod-docker]}, HostConfig={MemorySwap=-1, CPUShares=2, OomKillDisable=false, Privileged=false, PublishAllPorts=false, DNS=[192.168.192.117 192.168.192.117], DNSSearch=[default.svc.cluster.local svc.cluster.local cluster.local openstacklocal test.osp.sfa.se], NetworkMode=default, SecurityOpt=[label:level:s0:c1,c0], ReadonlyRootfs=false, LogConfig={Type:json-file Config:map[]}, ShmSize=67108864}}" May 17 17:58:20 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: 2016/05/17 17:58:20 http: multiple response.WriteHeader calls May 17 17:58:20 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:20.095609032+02:00" level=info msg="{Action=stop, ID=a74cfcb04b2c581cfa0464f34ef8e1549e90270c7df5791742671bfad0cae84a, LoginUID=4294967295, PID=107582}" May 17 17:58:45 infra-a2-mz0r8m82.test.osp.sfa.se forward-journal[14824]: time="2016-05-17T17:58:45.908218522+02:00" level=info msg="{Action=remove, ID=aabe1a1e3bf7b82caae86009ff77c3459cabfd97c0191d4bef24c56fa06ce107, LoginUID=4294967295, PID=107582}"
https://github.com/openshift/openshift-ansible/pull/1913
Changing component since this is an installer issue
(In reply to Marcel Wysocki from comment #7) > https://github.com/openshift/openshift-ansible/pull/1913 You've confirmed this fixes it?
yes. this approach fixes it and the other one mentioned in the PR.
https://github.com/openshift/openshift-ansible/pull/1941 provides a cleaner fix that works both when /usr/bin/docker-current exists and when it doesn't.
Check on openshift-ansible-3.0.93-1 1. New library dependency is added to the docker-1.10 package, It is not enough to bind the docker and docker-current file to container. # docker run -it -v /usr/bin/docker-current:/usr/bin/docker-current --entrypoint bash registry.access.redhat.com/openshift3/node:v3.2 [root@bc52ea75ba29 origin]# docker-current version docker-current: error while loading shared libraries: libseccomp.so.2: cannot open shared object file: No such file or directory 2. If user use ose-3.1 on the latest Atomic, we bind the docker binary to container. This means that openshift run with the docker-1.9 or newer version.
Dan, Can the SDN use docker from /rootfs whenever the env var OPENSHIFT_CONTAINERIZED == true rather than us trying to chase dependencies being mounted into the right places?
Ma, docker-1.10 is potentially not shipping but we'd definitely need to fix this if and when that happens. Are you ok with limiting the scope of this to only docker-1.9 which is currently supported maximum version? We should clone this.
Scott How to deal with the second issue? We still run ose-3.1.1.6 with docker-1.9 on Atomic.
I've opened https://bugzilla.redhat.com/show_bug.cgi?id=1340324 to address the long term problem. Lets limit the scope of this bug to only fixing containerized installs on docker-1.9.1.
OK, move this issue to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1208