1880514 – Etcd degraded during 4.5.9 to 4.6 nightly upgrade

Bug 1880514 - Etcd degraded during 4.5.9 to 4.6 nightly upgrade

Summary: Etcd degraded during 4.5.9 to 4.6 nightly upgrade

Keywords:
Status:	CLOSED DUPLICATE of bug 1880591
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Etcd
Sub Component:
Version:	4.6
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	high
Target Milestone:	---
Target Release:	4.7.0
Assignee:	Sam Batschelet
QA Contact:	ge liu
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1880512 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-09-18 17:43 UTC by Paige Rubendall
Modified:	2020-09-24 17:58 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-09-24 17:58:40 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Paige Rubendall 2020-09-18 17:43:59 UTC

Description of problem:
Stuck on degraded etcd cluster operator when upgrading from 4.5.9 to 4.6.0-0.nightly-2020-09-17-195238 

Version-Release number of selected component (if applicable): 4.6


How reproducible: 100%


Steps to Reproduce:
1. Create 4.5.9 cluster
2. Upgrade to 4.6.0-0.nightly-2020-09-17-195238


Actual results:
Unable to apply 4.6.0-0.nightly-2020-09-17-195238: the cluster operator etcd is degraded

Expected results:
Successful upgrade to 4.6.0-0.nightly-2020-09-17-195238 

Additional info:
#oc get co:NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
[2020-09-18T08:42:14.512Z] authentication                             4.6.0-0.nightly-2020-09-17-195238   True        False         True       84m
[2020-09-18T08:42:14.512Z] cloud-credential                           4.6.0-0.nightly-2020-09-17-195238   True        False         False      4h50m
[2020-09-18T08:42:14.512Z] cluster-autoscaler                         4.6.0-0.nightly-2020-09-17-195238   True        False         False      4h41m
[2020-09-18T08:42:14.512Z] config-operator                            4.6.0-0.nightly-2020-09-17-195238   True        False         False      4h41m
[2020-09-18T08:42:14.512Z] console                                    4.6.0-0.nightly-2020-09-17-195238   True        False         False      85m
[2020-09-18T08:42:14.512Z] csi-snapshot-controller                    4.6.0-0.nightly-2020-09-17-195238   True        False         False      4h23m
[2020-09-18T08:42:14.512Z] dns                                        4.6.0-0.nightly-2020-09-17-195238   True        False         True       87m
[2020-09-18T08:42:14.512Z] etcd                                       4.6.0-0.nightly-2020-09-17-195238   True        False         True       4h46m
[2020-09-18T08:42:14.512Z] image-registry                             4.6.0-0.nightly-2020-09-17-195238   True        False         False      89m
[2020-09-18T08:42:14.512Z] ingress                                    4.6.0-0.nightly-2020-09-17-195238   True        False         False      104m
[2020-09-18T08:42:14.512Z] insights                                   4.6.0-0.nightly-2020-09-17-195238   True        False         False      4h42m
[2020-09-18T08:42:14.512Z] kube-apiserver                             4.6.0-0.nightly-2020-09-17-195238   True        False         True       4h46m
[2020-09-18T08:42:14.512Z] kube-controller-manager                    4.6.0-0.nightly-2020-09-17-195238   True        False         True       4h46m
[2020-09-18T08:42:14.512Z] kube-scheduler                             4.6.0-0.nightly-2020-09-17-195238   True        False         True       4h45m
[2020-09-18T08:42:14.512Z] kube-storage-version-migrator              4.6.0-0.nightly-2020-09-17-195238   True        False         False      3h41m
[2020-09-18T08:42:14.513Z] machine-api                                4.6.0-0.nightly-2020-09-17-195238   True        False         False      4h42m
[2020-09-18T08:42:14.513Z] machine-approver                           4.6.0-0.nightly-2020-09-17-195238   True        False         False      4h45m
[2020-09-18T08:42:14.513Z] machine-config                             4.5.9                               False       True          True       87m
[2020-09-18T08:42:14.513Z] marketplace                                4.6.0-0.nightly-2020-09-17-195238   True        False         False      88m
[2020-09-18T08:42:14.513Z] monitoring                                 4.6.0-0.nightly-2020-09-17-195238   False       True          True       68m
[2020-09-18T08:42:14.513Z] network                                    4.6.0-0.nightly-2020-09-17-195238   True        True          True       4h48m
[2020-09-18T08:42:14.513Z] node-tuning                                4.6.0-0.nightly-2020-09-17-195238   True        False         False      104m
[2020-09-18T08:42:14.513Z] openshift-apiserver                        4.6.0-0.nightly-2020-09-17-195238   True        False         True       92m
[2020-09-18T08:42:14.513Z] openshift-controller-manager               4.6.0-0.nightly-2020-09-17-195238   True        False         False      103m
[2020-09-18T08:42:14.513Z] openshift-samples                          4.6.0-0.nightly-2020-09-17-195238   True        False         False      95m
[2020-09-18T08:42:14.513Z] operator-lifecycle-manager                 4.6.0-0.nightly-2020-09-17-195238   True        False         False      4h47m
[2020-09-18T08:42:14.513Z] operator-lifecycle-manager-catalog         4.6.0-0.nightly-2020-09-17-195238   True        False         False      4h47m
[2020-09-18T08:42:14.513Z] operator-lifecycle-manager-packageserver   4.6.0-0.nightly-2020-09-17-195238   True        False         False      85m
[2020-09-18T08:42:14.513Z] service-ca                                 4.6.0-0.nightly-2020-09-17-195238   True        False         False      4h48m
[2020-09-18T08:42:14.513Z] storage                                    4.6.0-0.nightly-2020-09-17-195238   True        False         False      104m


Name:         etcd
[2020-09-18T08:42:16.117Z] Namespace:    
[2020-09-18T08:42:16.117Z] Labels:       <none>
[2020-09-18T08:42:16.117Z] Annotations:  exclude.release.openshift.io/internal-openshift-hosted: true
[2020-09-18T08:42:16.117Z] API Version:  config.openshift.io/v1
[2020-09-18T08:42:16.117Z] Kind:         ClusterOperator
[2020-09-18T08:42:16.117Z] Metadata:
[2020-09-18T08:42:16.117Z]   Creation Timestamp:  2020-09-18T03:51:36Z
[2020-09-18T08:42:16.117Z]   Generation:          1
[2020-09-18T08:42:16.117Z]   Managed Fields:
[2020-09-18T08:42:16.117Z]     API Version:  config.openshift.io/v1
[2020-09-18T08:42:16.117Z]     Fields Type:  FieldsV1
[2020-09-18T08:42:16.117Z]     fieldsV1:
[2020-09-18T08:42:16.117Z]       f:metadata:
[2020-09-18T08:42:16.117Z]         f:annotations:
[2020-09-18T08:42:16.117Z]           .:
[2020-09-18T08:42:16.117Z]           f:exclude.release.openshift.io/internal-openshift-hosted:
[2020-09-18T08:42:16.117Z]       f:spec:
[2020-09-18T08:42:16.117Z]       f:status:
[2020-09-18T08:42:16.117Z]         .:
[2020-09-18T08:42:16.117Z]         f:extension:
[2020-09-18T08:42:16.117Z]         f:relatedObjects:
[2020-09-18T08:42:16.117Z]     Manager:      cluster-version-operator
[2020-09-18T08:42:16.117Z]     Operation:    Update
[2020-09-18T08:42:16.117Z]     Time:         2020-09-18T03:51:36Z
[2020-09-18T08:42:16.117Z]     API Version:  config.openshift.io/v1
[2020-09-18T08:42:16.117Z]     Fields Type:  FieldsV1
[2020-09-18T08:42:16.117Z]     fieldsV1:
[2020-09-18T08:42:16.117Z]       f:status:
[2020-09-18T08:42:16.117Z]         f:conditions:
[2020-09-18T08:42:16.117Z]         f:versions:
[2020-09-18T08:42:16.117Z]     Manager:         cluster-etcd-operator
[2020-09-18T08:42:16.117Z]     Operation:       Update
[2020-09-18T08:42:16.117Z]     Time:            2020-09-18T07:20:11Z
[2020-09-18T08:42:16.117Z]   Resource Version:  124315
[2020-09-18T08:42:16.117Z]   Self Link:         /apis/config.openshift.io/v1/clusteroperators/etcd
[2020-09-18T08:42:16.117Z]   UID:               f1e9fd95-5634-484c-bb77-d7a79cd72a30
[2020-09-18T08:42:16.117Z] Spec:
[2020-09-18T08:42:16.117Z] Status:
[2020-09-18T08:42:16.117Z]   Conditions:
[2020-09-18T08:42:16.117Z]     Last Transition Time:  2020-09-18T07:20:11Z
[2020-09-18T08:42:16.117Z]     Message:               NodeControllerDegraded: The master nodes not ready: node "ugdci18113952-prlsm-control-plane-0" not ready since 2020-09-18 07:19:04 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
[2020-09-18T08:42:16.117Z] EtcdMembersDegraded: 2 of 3 members are available, ugdci18113952-prlsm-control-plane-0 is unhealthy
[2020-09-18T08:42:16.117Z]     Reason:                EtcdMembers_UnhealthyMembers::NodeController_MasterNodesReady
[2020-09-18T08:42:16.117Z]     Status:                True
[2020-09-18T08:42:16.117Z]     Type:                  Degraded
[2020-09-18T08:42:16.117Z]     Last Transition Time:  2020-09-18T06:43:36Z
[2020-09-18T08:42:16.117Z]     Message:               NodeInstallerProgressing: 3 nodes are at revision 4
[2020-09-18T08:42:16.117Z] EtcdMembersProgressing: No unstarted etcd members found
[2020-09-18T08:42:16.117Z]     Reason:                AsExpected
[2020-09-18T08:42:16.117Z]     Status:                False
[2020-09-18T08:42:16.117Z]     Type:                  Progressing
[2020-09-18T08:42:16.117Z]     Last Transition Time:  2020-09-18T03:55:30Z
[2020-09-18T08:42:16.117Z]     Message:               StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 4
[2020-09-18T08:42:16.117Z] EtcdMembersAvailable: 2 of 3 members are available, ugdci18113952-prlsm-control-plane-0 is unhealthy
[2020-09-18T08:42:16.117Z]     Reason:                AsExpected
[2020-09-18T08:42:16.117Z]     Status:                True
[2020-09-18T08:42:16.117Z]     Type:                  Available
[2020-09-18T08:42:16.117Z]     Last Transition Time:  2020-09-18T03:53:57Z
[2020-09-18T08:42:16.117Z]     Reason:                AsExpected
[2020-09-18T08:42:16.117Z]     Status:                True
[2020-09-18T08:42:16.117Z]     Type:                  Upgradeable
[2020-09-18T08:42:16.117Z]   Extension:               <nil>
[2020-09-18T08:42:16.117Z]   Related Objects:
[2020-09-18T08:42:16.117Z]     Group:     operator.openshift.io
[2020-09-18T08:42:16.117Z]     Name:      cluster
[2020-09-18T08:42:16.117Z]     Resource:  etcds
[2020-09-18T08:42:16.117Z]     Group:     
[2020-09-18T08:42:16.117Z]     Name:      openshift-config
[2020-09-18T08:42:16.117Z]     Resource:  namespaces
[2020-09-18T08:42:16.117Z]     Group:     
[2020-09-18T08:42:16.117Z]     Name:      openshift-config-managed
[2020-09-18T08:42:16.117Z]     Resource:  namespaces
[2020-09-18T08:42:16.117Z]     Group:     
[2020-09-18T08:42:16.117Z]     Name:      openshift-etcd-operator
[2020-09-18T08:42:16.117Z]     Resource:  namespaces
[2020-09-18T08:42:16.117Z]     Group:     
[2020-09-18T08:42:16.117Z]     Name:      openshift-etcd
[2020-09-18T08:42:16.117Z]     Resource:  namespaces
[2020-09-18T08:42:16.117Z]   Versions:
[2020-09-18T08:42:16.117Z]     Name:     raw-internal
[2020-09-18T08:42:16.117Z]     Version:  4.6.0-0.nightly-2020-09-17-195238
[2020-09-18T08:42:16.117Z]     Name:     operator
[2020-09-18T08:42:16.117Z]     Version:  4.6.0-0.nightly-2020-09-17-195238
[2020-09-18T08:42:16.117Z]     Name:     etcd
[2020-09-18T08:42:16.117Z]     Version:  4.6.0-0.nightly-2020-09-17-195238

Comment 4 Sam Batschelet 2020-09-18 23:18:45 UTC

*** Bug 1880512 has been marked as a duplicate of this bug. ***

Comment 6 Paige Rubendall 2020-09-24 17:58:40 UTC

On trying to reproduce this error, I am seeing the same issue as a separate bug. Marking as duplicate

*** This bug has been marked as a duplicate of bug 1880591 ***

Note You need to log in before you can comment on or make changes to this bug.