Bug 2042210
| Summary: | Redis pod crashing with OOMKilled in ACM 2.4 running on OCP 4.8 | ||
|---|---|---|---|
| Product: | Red Hat Advanced Cluster Management for Kubernetes | Reporter: | Nikhil Gupta <ngupta> |
| Component: | Search / Analytics | Assignee: | Jorge Padilla <jpadilla> |
| Status: | CLOSED NEXTRELEASE | QA Contact: | Xiang Yin <xiyin> |
| Severity: | high | Docs Contact: | Mikela Dockery <mdockery> |
| Priority: | urgent | ||
| Version: | rhacm-2.4 | CC: | ashafi, clasohm, fdewaley, gekis, jpadilla, mheppler, mlele, mrobson, rpu, rspagnol, tuado, xdharmai |
| Target Milestone: | --- | Flags: | ashafi:
qe_test_coverage-
bot-tracker-sync: rhacm-2.4.z+ |
| Target Release: | rhacm-2.4.5 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-09-02 15:03:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Nikhil Gupta
2022-01-19 02:30:22 UTC
Thanks for the detailed description. The root cause is the Redis pod running out of memory. This topic explains how to increase the memory limits. https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.4/html/web_console/web-console#options-increase-memory The connection errors on the other search-* pods should recover once the Redis pod stabilizes, but you could also manually restart the pods to reestablish the connection. expending the memory limit wasn't enough, the connection errors continued as reported by the customer (raising to 10gb and then 16gb). could there be other changes to make? no logs were provided, so I did ask for a new set. it turns out that the PV what the Redis is using was full and there were no space left. By default, ACM installed with 10GB PV, and that was full after few months of running it. A new volume was created (the customer didn't see how to extend it dynamically), used different storage class and increased it to 50GB. no more crashes. do we have a better process I could use to create a kbase article, and how much storage would be require for a setup where ACM manags between 20 to 40 clusters? To prevent this problem from happening, the search operator should check that the PVC is larger than the memory limit. We should also update this topic. https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.4/html/web_console/web-console#options-increase-memory The guidance in this topic needs to be reviewed. We are seeing a larger number of resources per cluster, which increases the required memory. https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.4/html/install/installing#search-scalability From customer: > Compared to the secret creation date, could you provide a timeline of when ACM was first installed, when ACM was upgraded, and when this problem was noticed? Looking at the creation timestamps of a few resources, I think we originally installed RHACM 2.3 on 10/28/2021 (creation of namespaces open-cluster-management and open-cluster-management-hub), and we upgraded it to RHACM 2.4 on 1/5/2022 (creation of advanced-cluster-management.v2.4.1-* cluster roles). That would mean that the redisgraph-user-secret is the original one from when we installed RHACM. > Is there a reason why the search operator pod was created yesterday? We are working on automating post-install configuration items that include changes to machine config resources. Every time we deploy a new version of those, all worker nodes are rebooted, which causes most of the pods to be recreated. It sounds like the restart of the nodes is triggering this problem. I'd like to confirm that this. Also it would help to collect must-gather before and after the restart event. I'll try to recreate this scenario in another cluster. *** Bug 2047273 has been marked as a duplicate of this bug. *** @fdewaley Please ask the customer to run the debug script in comment 18. It would help us if they can capture 2 snapshots, first with the system stable and another one while observing high memory consumption. Also, it would help to capture must gather data around the same time. It's possible that some data is not getting deleted, the debug script should give us some indication if there's a leak. We have identified a couple factors contributing to the increased memory usage. 1. In some scenarios, saving and loading data from a PVC is too slow and requires too much memory. This completely negates the benefit from persisting the Redis state. In this case, turning off persistence can make the Redis instance more stable. More information in this bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2036197#c29 2. Sometimes when deleting a cluster from ACM, the data is not correctly removed from Redisgraph. This can happen if either search-redisgraph or the search-aggregator pods experience problems around the time that a cluster is removed from ACM. We have seen this problem in ACM environments where managed clusters are created and deleted often. We are working on a fix for this scenario. (In reply to Jorge Padilla from comment #27) > We have identified a couple factors contributing to the increased memory > usage. > > 1. In some scenarios, saving and loading data from a PVC is too slow and > requires too much memory. This completely negates the benefit from > persisting the Redis state. In this case, turning off persistence can make > the Redis instance more stable. More information in this bugzilla: > https://bugzilla.redhat.com/show_bug.cgi?id=2036197#c29 > > 2. Sometimes when deleting a cluster from ACM, the data is not correctly > removed from Redisgraph. This can happen if either search-redisgraph or the > search-aggregator pods experience problems around the time that a cluster is > removed from ACM. We have seen this problem in ACM environments where > managed clusters are created and deleted often. We are working on a fix for > this scenario. regarding point 2 is there a way to clear up the data. Aa a workaround to clear the Redis data for deleted clusters you could execute these queries directly in the search-redisgraph-0 pod.
This command lists all clusters for which Redisgraph has data.
---
oc exec -it search-redisgraph-0 -n open-cluster-management -- /bin/bash <<EOF
export REDISCLI_AUTH=\$REDIS_PASSWORD
redis-cli graph.query search-db "MATCH (n) RETURN DISTINCT n.cluster AS clusterName"
EOF
---
This command will delete data for clusterA and clusterB (replace with your actual cluster names)
---
oc exec -it search-redisgraph-0 -n open-cluster-management -- /bin/bash <<EOF
export REDISCLI_AUTH=\$REDIS_PASSWORD
redis-cli graph.query search-db "MATCH (n) WHERE n.cluster IN ['clusterA', 'clusterB'] DELETE n"
EOF
---
*** Bug 2052910 has been marked as a duplicate of this bug. *** G2Bsync 1124031852 comment crizzo71 Wed, 11 May 2022 17:11:55 UTC G2Bsync No code changes in 2.4.5 The solution is Search V2 Solution to this problem is addressed in next version of search feature. You can try out the DEV-PREVIEW operator here https://github.com/stolostron/search-v2-operator . Give it a shot and let us know your feedback. Plan is to include this in next major version of ACM. Closing this BZ as FUTURE target. *** Bug 2138423 has been marked as a duplicate of this bug. *** |