Bug 1592781
| Summary: | [upgrade] "remove /var/run/openshift-sdn: device or resource busy" while upgrading cri-o based cluster | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Weihua Meng <wmeng> |
| Component: | Networking | Assignee: | Casey Callendrello <cdc> |
| Status: | CLOSED DUPLICATE | QA Contact: | Meng Bo <bmeng> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.10.0 | CC: | aos-bugs, jokerman, mmccomas |
| Target Milestone: | --- | ||
| Target Release: | 3.10.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-06-19 13:37:35 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I imagine the OVS pod isn't running for some reason? Moving to networking. This is the same issue we had before: I restarted crio, and the node came back immediately. I'm fairly certain there is a race condition now. *** This bug has been marked as a duplicate of bug 1591752 *** |
Description of problem: "remove /var/run/openshift-sdn: device or resource busy" while upgrading cri-o based cluster Version-Release number of the following components: openshift-ansible-3.10.1-1.git.157.2bb6250.el7.noarch.rpm How reproducible: Steps to Reproduce: 1. upgrade OCP HA cri-o container runtime Actual results: Failure summary: 1. Hosts: qe-wmeng-upgrader5glb2-mez-2.0619-2xq.qe.rhcloud.com Play: Update master nodes Task: Wait for node to be ready Message: Failed without returning a message. [root@qe-wmeng-upgrader5glb2-mez-2 ~]# oc get node -o wide NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME qe-wmeng-upgrader5glb2-mez-1 Ready master 2h v1.10.0+b81c8f8 35.232.166.248 Red Hat Enterprise Linux Server 7.4 (Maipo) 3.10.0-693.21.1.el7.x86_64 cri-o://1.10.3 qe-wmeng-upgrader5glb2-mez-2 NotReady master 2h v1.10.0+b81c8f8 35.226.100.192 Red Hat Enterprise Linux Server 7.4 (Maipo) 3.10.0-693.21.1.el7.x86_64 cri-o://1.10.3 qe-wmeng-upgrader5glb2-mez-3 Ready master 2h v1.9.1+a0ce1bc657 130.211.128.175 Red Hat Enterprise Linux Server 7.4 (Maipo) 3.10.0-693.21.1.el7.x86_64 cri-o://1.9.12 qe-wmeng-upgrader5glb2-nrriz-1 Ready infra 2h v1.9.1+a0ce1bc657 104.198.34.215 Red Hat Enterprise Linux Server 7.4 (Maipo) 3.10.0-693.21.1.el7.x86_64 cri-o://1.9.12 qe-wmeng-upgrader5glb2-nrriz-2 Ready infra 2h v1.9.1+a0ce1bc657 35.225.14.150 Red Hat Enterprise Linux Server 7.4 (Maipo) 3.10.0-693.21.1.el7.x86_64 cri-o://1.9.12 qe-wmeng-upgrader5glb2-nzp-1 Ready compute 2h v1.9.1+a0ce1bc657 35.224.65.243 Red Hat Enterprise Linux Server 7.4 (Maipo) 3.10.0-693.21.1.el7.x86_64 cri-o://1.9.12 qe-wmeng-upgrader5glb2-nzp-2 Ready compute 2h v1.9.1+a0ce1bc657 35.188.103.37 Red Hat Enterprise Linux Server 7.4 (Maipo) 3.10.0-693.21.1.el7.x86_64 cri-o://1.9.12 The sdn pods running well [root@qe-wmeng-upgrader5glb2-mez-2 ~]# oc get pod -n openshift-sdn -o wide NAME READY STATUS RESTARTS AGE IP NODE ovs-2j6nr 1/1 Running 8 1h 10.240.0.23 qe-wmeng-upgrader5glb2-nrriz-1 ovs-4fqvp 1/1 Running 0 1h 10.240.0.27 qe-wmeng-upgrader5glb2-mez-2 ovs-9qxf6 1/1 Running 8 1h 10.240.0.25 qe-wmeng-upgrader5glb2-nzp-1 ovs-lk8zr 1/1 Running 8 1h 10.240.0.26 qe-wmeng-upgrader5glb2-nrriz-2 ovs-mp94v 1/1 Running 1 1h 10.240.0.30 qe-wmeng-upgrader5glb2-mez-1 ovs-sc8xt 1/1 Running 8 1h 10.240.0.28 qe-wmeng-upgrader5glb2-nzp-2 ovs-wt7bw 1/1 Running 8 1h 10.240.0.31 qe-wmeng-upgrader5glb2-mez-3 sdn-8ll8d 1/1 Running 8 1h 10.240.0.23 qe-wmeng-upgrader5glb2-nrriz-1 sdn-c4tf6 1/1 Running 8 1h 10.240.0.31 qe-wmeng-upgrader5glb2-mez-3 sdn-fjhbz 1/1 Running 8 1h 10.240.0.28 qe-wmeng-upgrader5glb2-nzp-2 sdn-gb64v 1/1 Running 8 1h 10.240.0.26 qe-wmeng-upgrader5glb2-nrriz-2 sdn-px9ls 1/1 Running 8 1h 10.240.0.25 qe-wmeng-upgrader5glb2-nzp-1 sdn-tvpd2 1/1 Running 0 34m 10.240.0.27 qe-wmeng-upgrader5glb2-mez-2 sdn-z9gvf 1/1 Running 0 1h 10.240.0.30 qe-wmeng-upgrader5glb2-mez-1 Errors observerd in the sdn pod where master node got NotReady E0619 09:01:41.536026 53582 cniserver.go:146] failed to remove old pod info socket: remove /var/run/openshift-sdn: device or resource busy E0619 09:01:41.536083 53582 cniserver.go:149] failed to remove contents of socket directory: remove /var/run/openshift-sdn: device or resource busy [root@qe-wmeng-upgrader5glb2-mez-2 ~]# ls /etc/cni/net.d/ 80-openshift-network.conf [root@qe-wmeng-upgrader5glb2-mez-2 ~]# cat /etc/cni/net.d/80-openshift-network.conf { "cniVersion": "0.2.0", "name": "openshift-sdn", "type": "openshift-sdn" } [root@qe-wmeng-upgrader5glb2-mez-2 ~]# ls -l /var/run/openshift-sdn/ total 4 srw-------. 1 root root 0 6月 19 05:01 cni-server.sock -r--r--r--. 1 root root 12 6月 19 05:01 config.json [root@qe-wmeng-upgrader5glb2-mez-2 ~]# ls -l /opt/cni/bin/ total 12836 -rwxr-xr-x. 1 root root 3079040 6月 19 05:01 host-local -rwxr-xr-x. 1 root root 3131888 6月 19 05:01 loopback -rwxr-xr-x. 1 root root 6927928 6月 19 05:01 openshift-sdn Expected results: Upgrade succeeds