Bug 2065318

Summary:	search-redisgraph-0 generating massive amount of logs after 2.4.2 upgrade
Product:	Red Hat Advanced Cluster Management for Kubernetes	Reporter:	James Young <jayoung>
Component:	Search / Analytics	Assignee:	Jorge Padilla <jpadilla>
Status:	CLOSED ERRATA	QA Contact:	Atif <ashafi>
Severity:	unspecified	Docs Contact:	Mikela Dockery <mdockery>
Priority:	unspecified
Version:	rhacm-2.4	CC:	dbennett, jpadilla, xdharmai
Target Milestone:	---	Flags:	bot-tracker-sync: rhacm-2.4.z+
Target Release:	rhacm-2.4.5
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-06-27 17:04:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 4 Jorge Padilla 2022-03-17 16:48:48 UTC

It looks like certificates got out of sync. For a quick workaround, let's try restarting the search-aggregator pod, if that doesn't work then also restart search-redisgraph and search-api.  If restarting works it will confirm the root cause.

For a permanent fix we need to find out why this happened.
Did any nodes in the cluster got restarted recently?
Did anything else happened that could have triggered a certificate rotation?

Comment 8 Jorge Padilla 2022-05-03 17:28:43 UTC

I've merged 2 changes for ACM 2.5.  Waiting to merge into 2.4.5

1. Fixed the liveness probe to check for redis on port 6379 instead of the stunnel port 6380.
2. Reduced the default log level from 7 (all) to 4 (warning)

Comment 9 Jorge Padilla 2022-05-06 16:36:33 UTC

Fix has been merged for 2.4.5 and 2.5.0

Changes:
1. Fixed the liveness probe to check for redis on port 6379 instead of the stunnel port 6380.
2. Reduced the default log level from 7 (all) to 4 (warning)

Errata doc:
Search becomes unavailable and can't recover if there's a failure in Redis. It causes a large volume of error logs. This change fixes the health monitoring allowing the search component to recover from this state and reduces the log verbosity.

Comment 10 Atif 2022-06-08 21:10:55 UTC

Verified on v2.4.5-FC0.

Comment 16 errata-xmlrpc 2022-06-27 17:04:01 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Advanced Cluster Management 2.4.5 security updates and bug fixes), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5201