Bug 1836299
| Summary: | NooBaa Operator deploys with HPA that fires maxreplicas alerts by default | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Container Storage | Reporter: | Caden Marchese <cmarches> |
| Component: | Multi-Cloud Object Gateway | Assignee: | Ohad <omitrani> |
| Status: | CLOSED ERRATA | QA Contact: | Filip Balák <fbalak> |
| Severity: | low | Docs Contact: | Erin Donnelly <edonnell> |
| Priority: | unspecified | ||
| Version: | 4.3 | CC: | aivaraslaimikis, aos-bugs, assingh, bkunal, ebondare, edonnell, etamir, fbalak, jritter, jsafrane, madam, mbukatov, mpandey, muagarwa, nbecker, nberry, ocs-bugs, omitrani, ratamir, sostapov, tdesala |
| Target Milestone: | --- | ||
| Target Release: | OCS 4.6.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | v4.6.0-56.ci | Doc Type: | Bug Fix |
| Doc Text: |
.`MAX HPA` value exceeding `1` no longer triggers an alert
In previous versions of Red Hat OpenShift Container Storage, the autoscaling feature for pods was not available. Therefore, the `MAX HPA` value could not be greater than `1`, or an alert was triggered. With this update, this feature is enabled and the alert is no longer triggered.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-12-17 06:22:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1826482, 1859307, 1882359 | ||
Moving to OCS (and guessing OCS version an component name). @raz I guess we can push out of 4.5 since auto scale won't be in ? *** Bug 1800599 has been marked as a duplicate of this bug. *** *** Bug 1840102 has been marked as a duplicate of this bug. *** @Bipin & @Nimrod, Is this the epic for the auto-scale of noobaa-endpoint? https://issues.redhat.com/browse/KNIP-1422 Regards, Ashish Singh Yes. As mentioned before, as well as another investigation on a similar bug (See https://bugzilla.redhat.com/show_bug.cgi?id=1788126#c23) These alerts cannot be suppressed. Pushing to 4.6 when autoscaling might be delivered and then those alerts would stop. (In reply to Nimrod Becker from comment #21) > As mentioned before, as well as another investigation on a similar bug (See > https://bugzilla.redhat.com/show_bug.cgi?id=1788126#c23) > These alerts cannot be suppressed. BZ 1788126 could be fixed via inhibition rules, as noted in comment https://bugzilla.redhat.com/show_bug.cgi?id=1788126#c29 Please reevaluate. If you want to claim that this can't be fixed, I expect mcg dev team to find someone from openshift team to validate such opinion. This can't be fixed, we need to change the scale of HPA to 1-2 which will happen in 4.6 As part of the endpoint HPA TP in 4.6, the default was set to 1 - 2, the alerts won't fire. Closing this. It seems that the fix was not complete.
I don't see alert mentioned in bug description:
"HPA openshift-storage/noobaa-endpoint has been running at max replicas for longer than 15 minutes."
but I still see the following alert after installation:
"KubeHpaReplicasMismatch: HPA openshift-storage/noobaa-endpoint has not matched the desired number of replicas for longer than 15 minutes."
-> ASSIGNED
Tested with:
ocs-operator.v4.6.0-108.ci
$ oc get deployment noobaa-endpoint -n openshift-storage -o yaml
kind: Deployment
apiVersion: apps/v1
(...)
spec:
replicas: 1
(...)
status:
observedGeneration: 1
replicas: 1
updatedReplicas: 1
readyReplicas: 1
availableReplicas: 1
$ oc get HorizontalPodAutoscaler noobaa-endpoint -n openshift-storage -o yaml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
(...)
spec:
maxReplicas: 2
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: noobaa-endpoint
targetCPUUtilizationPercentage: 80
status:
currentReplicas: 1
desiredReplicas: 0
@Filip, I can tell from the last message that this is not the same issue. Can we please close this one and open a new one for the "desiredReplicas: 0" problem? Ok, based on comments 28 and 29 I VERIFY that the original HPA alert is gone. For remaining HPA problems were reported BZ 1885313 and BZ 1885320. Nimrod/Ohad, I have changed the doc type to 'Bug Fix' because the BZ is fixed now, pls provide the doc text accordingly. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenShift Container Storage 4.6.0 security, bug fix, enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5605 |
Description of problem: Deploying OCS rolls out a NooBaa operator that creates the following HPA: apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler ... spec: maxReplicas: 1 minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: noobaa-endpoint targetCPUUtilizationPercentage: 80 status: currentCPUUtilizationPercentage: 1 currentReplicas: 1 desiredReplicas: 1 This HPA results in the below alert: May 6, 8:56 am HPA openshift-storage/noobaa-endpoint has been running at max replicas for longer than 15 minutes. Since the maxReplicas and minReplicas are 1, it will always be at max replicas, and any attempt to disable the HPA or expand max replicas is overwritten, so it seems like a potential defect. Version-Release number of selected component (if applicable): 4.3.18 Steps to Reproduce: 1. Deploy OCS on top of 4.3. 2. Wait for NooBaa operator to deploy. 3. Observe the HPA alerts. Actual results: Alert fires Expected results: I assume that the HPA is there for some reason, but if there's no ability to change it without editing the operator and there is no scaleable range, it could just be removed from the operator. I can provide must-gather logs on request, if they are needed.