Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1720978

Summary: [3.10] Upgrade failed at Task: Wait for node to be ready
Product: OpenShift Container Platform Reporter: Weihua Meng <wmeng>
Component: Cluster Version OperatorAssignee: Joseph Callen <jcallen>
Status: CLOSED ERRATA QA Contact: Weihua Meng <wmeng>
Severity: high Docs Contact:
Priority: high    
Version: 3.10.0CC: aos-bugs, cdc, jcallen, jokerman, mmccomas, padillon, vlaad, wsun
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: When openshift facts was recently modified the ipv4 dictionary item no longer existed Consequence: MTU was set incorrectly Fix: Remove the conditional with ipv4 Result: MTU set correctly.
Story Points: ---
Clone Of:
: 1730736 (view as bug list) Environment:
Last Closed: 2019-07-24 13:47:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1730736    

Description Weihua Meng 2019-06-17 03:30:47 UTC
Description of problem:
[3.10] Upgrade failed at Task: Wait for node to be ready

Version-Release number of the following components:
openshift-ansible-3.10.149-1.git.0.eb0262c.el7

How reproducible:
1 out of 1

Steps to Reproduce:
1. upgrade OCP v3.9.78 HA cluster to v3.10

Actual results:
Upgrade failed.

Failure summary:


  1. Hosts:    wmengahha3978-node-zone1-primary-1.0614-r5z.qe.rhcloud.com
     Play:     Drain and upgrade nodes
     Task:     Wait for node to be ready
     Message:  Failed without returning a message.

Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.429541     450 cni.go:279] Error deleting network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin]
Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.431586     450 remote_runtime.go:115] StopPodSandbox "f5969a5df560415ff0c0e8a8e23cc7310045f4354dbd72dffea350af772b5a1f" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "nodejs-mongodb-example-2-rf2kj_install-test" network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin]
Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.431635     450 kuberuntime_gc.go:153] Failed to stop sandbox "f5969a5df560415ff0c0e8a8e23cc7310045f4354dbd72dffea350af772b5a1f" before removing: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "nodejs-mongodb-example-2-rf2kj_install-test" network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin]
Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: W0616 14:28:21.436570     450 cni.go:242] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "6e715302e60bda7fdb0983cfb3a8177b443b128a2cb83a65c2110b59f0581a4d"
Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.436689     450 cni.go:279] Error deleting network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin]
Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.438653     450 remote_runtime.go:115] StopPodSandbox "6e715302e60bda7fdb0983cfb3a8177b443b128a2cb83a65c2110b59f0581a4d" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "mongodb-2-qhtg8_install-test" network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin]
Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.438697     450 kuberuntime_gc.go:153] Failed to stop sandbox


Expected results:
Upgrade succeed.

Comment 11 Joseph Callen 2019-06-20 18:52:44 UTC
Probably related to MTU issue.  Once https://github.com/openshift/openshift-ansible/pull/11707 is merged will cherry pick for previous versions.

Comment 15 Weihua Meng 2019-07-17 00:30:00 UTC
upgrade success with 1 master + 1 node cluster, RHEL Atomic Host

Comment 19 errata-xmlrpc 2019-07-24 13:47:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1755