It looks like certificates got out of sync. For a quick workaround, let's try restarting the search-aggregator pod, if that doesn't work then also restart search-redisgraph and search-api. If restarting works it will confirm the root cause. For a permanent fix we need to find out why this happened. Did any nodes in the cluster got restarted recently? Did anything else happened that could have triggered a certificate rotation?
I've merged 2 changes for ACM 2.5. Waiting to merge into 2.4.5 1. Fixed the liveness probe to check for redis on port 6379 instead of the stunnel port 6380. 2. Reduced the default log level from 7 (all) to 4 (warning)
Fix has been merged for 2.4.5 and 2.5.0 Changes: 1. Fixed the liveness probe to check for redis on port 6379 instead of the stunnel port 6380. 2. Reduced the default log level from 7 (all) to 4 (warning) Errata doc: Search becomes unavailable and can't recover if there's a failure in Redis. It causes a large volume of error logs. This change fixes the health monitoring allowing the search component to recover from this state and reduces the log verbosity.
Verified on v2.4.5-FC0.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Advanced Cluster Management 2.4.5 security updates and bug fixes), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5201