Bug 1946607

Summary:	etcd readinessProbe is not reflective of actual readiness
Product:	OpenShift Container Platform	Reporter:	Sam Batschelet <sbatsche>
Component:	Etcd	Assignee:	Sam Batschelet <sbatsche>
Status:	CLOSED ERRATA	QA Contact:	ge liu <geliu>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.7	CC:	david.karlsen, rhowe, wking, wlewis
Target Milestone:	---
Target Release:	4.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: The introduction of SO_REUSEADDR socket options has created a condition where the readiness probe for etcd does not actually properly reflecting the readiness of the operand. Consequence: etcd pod shows ready while quorum-guard fails. Fix: improve readinessProbe checks by dialing unix socket target Result: readiness probe for etcd actually properly reflecting the readiness of the operand.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-07-27 22:57:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sam Batschelet 2021-04-06 14:03:17 UTC

Description of problem: The introduction of SO_REUSEADDR socket options has created a condition where the readiness probe[1] for etcd does not actually properly reflecting the readiness of the operand.

In the etcd static pod we have code that blocks until a port is released[2]. Before SO_REUSEADDR this blocking allowed the simple TCP probe for readiness to be accurate. But this is no longer the case.


[1] https://github.com/openshift/cluster-etcd-operator/blob/release-4.8/bindata/etcd/pod.yaml#L165
[2] https://github.com/openshift/cluster-etcd-operator/blob/release-4.8/bindata/etcd/pod.yaml#L135

Version-Release number of selected component (if applicable):


How reproducible: 95%


Steps to Reproduce:
1. force new static pod revision `oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge 
`
2. watch pods in openshift-etcd namespace. `watch oc get pods -n openshift-etcd`
3. when a new static pod rolls out there will be a time where the etcd pod containers will show 3/3 ready yet health checks fail.

Actual results: etcd pod shows ready while quorum-guard fails.


Expected results: etcd readiness is reflective of operand health.


Additional info:

Comment 3 Sam Batschelet 2021-05-07 14:43:48 UTC

*** Bug 1928874 has been marked as a duplicate of this bug. ***

Comment 7 errata-xmlrpc 2021-07-27 22:57:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438