Bug 1885320
| Summary: | HPA fires KubeHpaReplicasMismatch alert after OCS installation | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Filip Balák <fbalak> |
| Component: | Node | Assignee: | Joel Smith <joelsmith> |
| Node sub component: | Autoscaler (HPA, VPA) | QA Contact: | Sunil Choudhary <schoudha> |
| Status: | CLOSED NOTABUG | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | alegrand, anpicker, aos-bugs, erooth, jokerman, kakkoyun, lcosic, mloibl, nagrawal, nberry, omitrani, pkrupa, rphillips, surbania |
| Version: | 4.6 | Flags: | joelsmith:
needinfo-
|
| Target Milestone: | --- | ||
| Target Release: | 4.7.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-10-15 12:12:39 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1885313 | ||
|
Description
Filip Balák
2020-10-05 15:27:25 UTC
reassigning to Node component who is handling HPA. Hi, I investigated the original issue with @Filip I would like to add the folowing information: 1. The KubeHpaReplicasMismatch event on the HPA is persistent and continues 2. From my obervation the cause for the event is fact that the HPA decided that the desiredReplicas should be 0 (we can see it in the status section of the HPA resource) while the actual replica count is 1 3. The desried replica should never be 0 because the spec specify minReplicas of 1 (and a maxReplicas of 2) 4. The current CPU metric observed by the HPA controller is "Unknown/80%", this can be seen when using the describe command on the HPA resource 5. We are aware there is a known issue regarding pods with initContainers (https://bugzilla.redhat.com/show_bug.cgi?id=1867477) but our pods' spec does not define any initContainer or any sidecar containers 6. This was testetd on top of OCP 4.6 we had a different issue when we tested on top of OCP 4.5 I hope this information is helpful Can you oc describe the hpa resource? Does the spec for openshift-storage/noobaa-endpoint contain a zero for the Replicas count? Setting it to zero will disable the autoscaling for that resource. Can you add the logs from the HPA as well? > Can you oc describe the hpa resource?
> Can you add the logs from the HPA as well?
@Filip Can you please provide it? maybe from the OCS bug?
@Ryan
The noobaa-endpoint deployment is created with a replica of 1 and we do not change it at all, we leave this responsibility to the HPA
Are you ok if we target a fix for this bug in 4.6.z? One more thing to keep in mind is after 1) Install OCP. Monitoring is usually still installing after an OCP cluster is 'up'. Does the ocs-operator make sure that all it's dependent components are running? > Does the ocs-operator make sure that all it's dependent components are running?
This is a very good point, and I am not sure what the answer is.
I will have to check and come back with an answer.
But even if this is correct, why does it not resolve itself later in the lifetime of the cluster?
|