Bug 1946607 - etcd readinessProbe is not reflective of actual readiness
Summary: etcd readinessProbe is not reflective of actual readiness
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Etcd
Version: 4.7
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.8.0
Assignee: Sam Batschelet
QA Contact: ge liu
URL:
Whiteboard:
: 1928874 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-06 14:03 UTC by Sam Batschelet
Modified: 2021-10-13 15:56 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The introduction of SO_REUSEADDR socket options has created a condition where the readiness probe for etcd does not actually properly reflecting the readiness of the operand. Consequence: etcd pod shows ready while quorum-guard fails. Fix: improve readinessProbe checks by dialing unix socket target Result: readiness probe for etcd actually properly reflecting the readiness of the operand.
Clone Of:
Environment:
Last Closed: 2021-07-27 22:57:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-etcd-operator pull 580 0 None open Bug 1946607: bindata/etcd: improve readinessProbe checks by dialing unix socket target 2021-04-30 18:22:09 UTC
Red Hat Knowledge Base (Solution) 5801821 0 None None None 2021-05-07 14:43:48 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:58:04 UTC

Description Sam Batschelet 2021-04-06 14:03:17 UTC
Description of problem: The introduction of SO_REUSEADDR socket options has created a condition where the readiness probe[1] for etcd does not actually properly reflecting the readiness of the operand.

In the etcd static pod we have code that blocks until a port is released[2]. Before SO_REUSEADDR this blocking allowed the simple TCP probe for readiness to be accurate. But this is no longer the case.


[1] https://github.com/openshift/cluster-etcd-operator/blob/release-4.8/bindata/etcd/pod.yaml#L165
[2] https://github.com/openshift/cluster-etcd-operator/blob/release-4.8/bindata/etcd/pod.yaml#L135

Version-Release number of selected component (if applicable):


How reproducible: 95%


Steps to Reproduce:
1. force new static pod revision `oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge 
`
2. watch pods in openshift-etcd namespace. `watch oc get pods -n openshift-etcd`
3. when a new static pod rolls out there will be a time where the etcd pod containers will show 3/3 ready yet health checks fail.

Actual results: etcd pod shows ready while quorum-guard fails.


Expected results: etcd readiness is reflective of operand health.


Additional info:

Comment 3 Sam Batschelet 2021-05-07 14:43:48 UTC
*** Bug 1928874 has been marked as a duplicate of this bug. ***

Comment 7 errata-xmlrpc 2021-07-27 22:57:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.