Bug 1720978 - [3.10] Upgrade failed at Task: Wait for node to be ready
Summary: [3.10] Upgrade failed at Task: Wait for node to be ready
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.10.z
Assignee: Joseph Callen
QA Contact: Weihua Meng
URL:
Whiteboard:
Depends On:
Blocks: 1730736
TreeView+ depends on / blocked
 
Reported: 2019-06-17 03:30 UTC by Weihua Meng
Modified: 2019-07-24 13:47 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: When openshift facts was recently modified the ipv4 dictionary item no longer existed Consequence: MTU was set incorrectly Fix: Remove the conditional with ipv4 Result: MTU set correctly.
Clone Of:
: 1730736 (view as bug list)
Environment:
Last Closed: 2019-07-24 13:47:19 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:1755 0 None None None 2019-07-24 13:47:24 UTC

Description Weihua Meng 2019-06-17 03:30:47 UTC
Description of problem:
[3.10] Upgrade failed at Task: Wait for node to be ready

Version-Release number of the following components:
openshift-ansible-3.10.149-1.git.0.eb0262c.el7

How reproducible:
1 out of 1

Steps to Reproduce:
1. upgrade OCP v3.9.78 HA cluster to v3.10

Actual results:
Upgrade failed.

Failure summary:


  1. Hosts:    wmengahha3978-node-zone1-primary-1.0614-r5z.qe.rhcloud.com
     Play:     Drain and upgrade nodes
     Task:     Wait for node to be ready
     Message:  Failed without returning a message.

Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.429541     450 cni.go:279] Error deleting network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin]
Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.431586     450 remote_runtime.go:115] StopPodSandbox "f5969a5df560415ff0c0e8a8e23cc7310045f4354dbd72dffea350af772b5a1f" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "nodejs-mongodb-example-2-rf2kj_install-test" network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin]
Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.431635     450 kuberuntime_gc.go:153] Failed to stop sandbox "f5969a5df560415ff0c0e8a8e23cc7310045f4354dbd72dffea350af772b5a1f" before removing: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "nodejs-mongodb-example-2-rf2kj_install-test" network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin]
Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: W0616 14:28:21.436570     450 cni.go:242] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "6e715302e60bda7fdb0983cfb3a8177b443b128a2cb83a65c2110b59f0581a4d"
Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.436689     450 cni.go:279] Error deleting network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin]
Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.438653     450 remote_runtime.go:115] StopPodSandbox "6e715302e60bda7fdb0983cfb3a8177b443b128a2cb83a65c2110b59f0581a4d" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "mongodb-2-qhtg8_install-test" network: failed to find plugin "portmap" in path [/opt/bridge/bin /opt/cni/bin]
Jun 16 14:28:21 wmengahha3978-node-zone1-primary-1 atomic-openshift-node[440]: E0616 14:28:21.438697     450 kuberuntime_gc.go:153] Failed to stop sandbox


Expected results:
Upgrade succeed.

Comment 11 Joseph Callen 2019-06-20 18:52:44 UTC
Probably related to MTU issue.  Once https://github.com/openshift/openshift-ansible/pull/11707 is merged will cherry pick for previous versions.

Comment 15 Weihua Meng 2019-07-17 00:30:00 UTC
upgrade success with 1 master + 1 node cluster, RHEL Atomic Host

Comment 19 errata-xmlrpc 2019-07-24 13:47:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1755


Note You need to log in before you can comment on or make changes to this bug.