Description of problem: When build pod complete/failed, there is an event show TeardownNetworkError: "Failed to teardown network for pod...... Version-Release number of selected component (if applicable): openshift v3.2.0.9 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 How reproducible: always Steps to Reproduce: 1.create a project 2.create a build $oc new-app eap64-basic-s2i 3.after the build pod complete check the build event $oc describe pod Actual results: [root@openshift-134 ~]# oc get pod NAME READY STATUS RESTARTS AGE eap-app-1-build 0/1 Completed 0 56m [root@openshift-134 ~]# oc describe pod eap-app-1-build Name: eap-app-1-build Namespace: haowang Node: openshift-126.lab.sjc.redhat.com/10.14.6.126 Start Time: Thu, 31 Mar 2016 01:42:11 -0400 Labels: openshift.io/build.name=eap-app-1 Status: Succeeded IP: 10.2.0.5 Controllers: <none> Containers: sti-build: Container ID: docker://0bad569d35ca9a3c0b3ae27fcca25c295b5a2a270a09c734c7851f5196b15956 Image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-sti-builder:v3.2.0.9 Image ID: docker://221a9942965715c5efc6192d45206a12eb797938a23adc788414c5f71cd907ab Port: Args: --loglevel=5 QoS Tier: cpu: BestEffort memory: BestEffort State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 31 Mar 2016 01:42:14 -0400 Finished: Thu, 31 Mar 2016 01:55:21 -0400 Ready: False Restart Count: 0 Environment Variables: BUILD: {"kind":"Build","apiVersion":"v1","metadata":{"name":"eap-app-1","namespace":"haowang","selfLink":"/oapi/v1/namespaces/haowang/builds/eap-app-1","uid":"50652e97-f703-11e5-a98d-fa163e306795","resourceVersion":"16153","creationTimestamp":"2016-03-31T05:42:11Z","labels":{"app":"eap-app","application":"eap-app","buildconfig":"eap-app","openshift.io/build-config.name":"eap-app","template":"eap64-basic-s2i","xpaas":"1.2.0"},"annotations":{"openshift.io/build.number":"1"}},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/jboss-developer/jboss-eap-quickstarts","ref":"6.4.x"},"contextDir":"kitchensink","secrets":null},"strategy":{"type":"Source","sourceStrategy":{"from":{"kind":"DockerImage","name":"registry.access.redhat.com/jboss-eap-6/eap64-openshift:1.2"},"forcePull":true}},"output":{"to":{"kind":"DockerImage","name":"172.31.48.195:5000/haowang/eap-app:latest"},"pushSecret":{"name":"builder-dockercfg-up2i6"}},"resources":{},"postCommit":{}},"status":{"phase":"New","outputDockerImageReference":"172.31.48.195:5000/haowang/eap-app:latest","config":{"kind":"BuildConfig","namespace":"haowang","name":"eap-app"}}} BUILD_LOGLEVEL: 5 SOURCE_REPOSITORY: https://github.com/jboss-developer/jboss-eap-quickstarts SOURCE_URI: https://github.com/jboss-developer/jboss-eap-quickstarts SOURCE_CONTEXT_DIR: kitchensink SOURCE_REF: 6.4.x ORIGIN_VERSION: v3.2.0.9 ALLOWED_UIDS: 1- DROP_CAPS: KILL,MKNOD,SETGID,SETUID,SYS_CHROOT PUSH_DOCKERCFG_PATH: /var/run/secrets/openshift.io/push Conditions: Type Status Ready False Volumes: docker-socket: Type: HostPath (bare host directory volume) Path: /var/run/docker.sock builder-dockercfg-up2i6-push: Type: Secret (a volume populated by a Secret) SecretName: builder-dockercfg-up2i6 builder-token-k6g87: Type: Secret (a volume populated by a Secret) SecretName: builder-token-k6g87 Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 14m 14m 1 {default-scheduler } Normal Scheduled Successfully assigned eap-app-1-build to openshift-126.lab.sjc.redhat.com 14m 14m 1 {kubelet openshift-126.lab.sjc.redhat.com} spec.containers{sti-build} Normal Pulled Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-sti-builder:v3.2.0.9" already present on machine 14m 14m 1 {kubelet openshift-126.lab.sjc.redhat.com} spec.containers{sti-build} Normal Created Created container with docker id 0bad569d35ca 14m 14m 1 {kubelet openshift-126.lab.sjc.redhat.com} spec.containers{sti-build} Normal Started Started container with docker id 0bad569d35ca 1m 1m 1 {kubelet openshift-126.lab.sjc.redhat.com} Warning FailedSync Error syncing pod, skipping: failed to "TeardownNetwork" for "eap-app-1-build_haowang" with TeardownNetworkError: "Failed to teardown network for pod \"50684028-f703-11e5-a98d-fa163e306795\" using network plugins \"redhat/openshift-ovs-multitenant\": exit status 1" Expected results: Teardown should success without errors Additional info: It seems that when teardown pod network, the host veth already gone , don't know why this happend,maybe already teardown before ? ... error log stack from: https://github.com/kubernetes/kubernetes/blob/b1cd74bd34c7a603599f55dafbdd05caad6821af/pkg/kubelet/dockertools/manager.go#L1295 https://github.com/openshift/openshift-sdn/blob/cf507f195b3c6e1f2fafc980ebe8656f3089349e/plugins/osdn/ovs/plugin.go#L200-L205 https://github.com/openshift/openshift-sdn/blob/cf507f195b3c6e1f2fafc980ebe8656f3089349e/plugins/osdn/ovs/bin/openshift-sdn-ovs#L60-L62 error log from node: Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: I0330 23:16:56.101432 5456 kubelet.go:2430] SyncLoop (PLEG): "frontend-2-hook-pre_xiuwang1(fac569e7-f6ee-11e5-a98d-fa163e306795)", Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: I0330 23:16:56.177431 5456 plugin.go:205] TearDownPod network plugin output: + lock_file=/var/lock/openshift-sdn.lock Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + action=teardown Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + net_container=b8cfbbd39700a92eb821b09e2767cf22ac6771e3e16df343cf8ef4d229cbe860 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + tenant_id=-1 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + ingress_bw=-1 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + egress_bw=-1 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + mtu=-1 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + lockwrap run Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + flock 200 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + run Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + get_ipaddr_pid_veth Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ docker inspect --format '{{.HostConfig.NetworkMode}}' b8cfbbd39700a92eb821b09e2767cf22ac6771e3e16df343cf8ef4d229cbe860 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + network_mode=default Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + '[' default == host ']' Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + [[ default =~ container:.* ]] Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ docker inspect --format '{{.NetworkSettings.IPAddress}}' b8cfbbd39700a92eb821b09e2767cf22ac6771e3e16df343cf8ef4d229cbe860 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + ipaddr= Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ docker inspect --format '{{.State.Pid}}' b8cfbbd39700a92eb821b09e2767cf22ac6771e3e16df343cf8ef4d229cbe860 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + pid=0 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ get_veth_host 0 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ local pid=0 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: +++ nsenter -n -t 0 -- ethtool -S eth0 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: +++ sed -n -e 's/.*peer_ifindex: //p' Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: nsenter: neither filename nor target pid supplied for ns/net Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ local veth_ifindex= Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ ip link show Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ sed -ne 's/^: \([^:@]*\).*/\1/p' Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + veth_host= Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ get_container_mac 0 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ local pid=0 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ nsenter -n -t 0 -- ip link show dev eth0 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ sed -n -e 's/.*link.ether \([^ ]*\).*/\1/p' Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: nsenter: neither filename nor target pid supplied for ns/net Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + macaddr= Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + source /run/openshift-sdn/config.env Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ export OPENSHIFT_CLUSTER_SUBNET=10.2.0.0/16 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ++ OPENSHIFT_CLUSTER_SUBNET=10.2.0.0/16 Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + case "$action" in Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + del_ovs_port Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: + ovs-vsctl --if-exists del-port Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: ovs-vsctl: 'del-port' command requires at least 1 arguments Mar 30 23:16:56 openshift-126.lab.sjc.redhat.com atomic-openshift-node[5456]: , exit status 1
*** This bug has been marked as a duplicate of bug 1322077 ***