Description of problem: [3.10] Upgrade failed at Task: Wait for node to be ready Version-Release number of the following components: openshift-ansible-3.10.149-1.git.0.eb0262c.el7 How reproducible: 1 out of 1 Steps to Reproduce: 1. upgrade OCP v3.9.78 HA cluster to v3.10 Actual results: Upgrade failed. Failure summary: 1. Hosts: wmengahha3978-node-zone1-primary-1.0614-r5z.qe.rhcloud.com Play: Drain and upgrade nodes Task: Wait for node to be ready Message: Failed without returning a message. Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.429541 450 cni.go:279] Error deleting network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin] Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.431586 450 remote_runtime.go:115] StopPodSandbox "f5969a5df560415ff0c0e8a8e23cc7310045f4354dbd72dffea350af772b5a1f" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "nodejs-mongodb-example-2-rf2kj_install-test" network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin] Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.431635 450 kuberuntime_gc.go:153] Failed to stop sandbox "f5969a5df560415ff0c0e8a8e23cc7310045f4354dbd72dffea350af772b5a1f" before removing: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "nodejs-mongodb-example-2-rf2kj_install-test" network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin] Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: W0616 14:28:21.436570 450 cni.go:242] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "6e715302e60bda7fdb0983cfb3a8177b443b128a2cb83a65c2110b59f0581a4d" Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.436689 450 cni.go:279] Error deleting network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin] Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.438653 450 remote_runtime.go:115] StopPodSandbox "6e715302e60bda7fdb0983cfb3a8177b443b128a2cb83a65c2110b59f0581a4d" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "mongodb-2-qhtg8_install-test" network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin] Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.438697 450 kuberuntime_gc.go:153] Failed to stop sandbox Expected results: Upgrade succeed.
Probably related to MTU issue. Once https://github.com/openshift/openshift-ansible/pull/11707 is merged will cherry pick for previous versions.
3.9 PR: https://github.com/openshift/openshift-ansible/pull/11709 3.10 PR: https://github.com/openshift/openshift-ansible/pull/11708
upgrade success with 1 master + 1 node cluster, RHEL Atomic Host
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1755