Bug 1751602
| Summary: | [4.1.z]monitoring reports Degraded after scaling up/down prometheus-k8s pods which bound with PVs | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> | ||||||||
| Component: | Monitoring | Assignee: | Pawel Krupa <pkrupa> | ||||||||
| Status: | CLOSED NOTABUG | QA Contact: | Junqi Zhao <juzhao> | ||||||||
| Severity: | low | Docs Contact: | |||||||||
| Priority: | low | ||||||||||
| Version: | 4.1.z | CC: | alegrand, anpicker, erooth, kakkoyun, lcosic, mloibl, pkrupa, surbania | ||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | 4.3.0 | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | wip | ||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | |||||||||||
| : | 1751607 (view as bug list) | Environment: | |||||||||
| Last Closed: | 2019-10-16 08:47:54 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | |||||||||||
| Bug Blocks: | 1751607 | ||||||||||
| Attachments: |
|
||||||||||
Created attachment 1614389 [details]
monitoring dump
Created attachment 1626292 [details]
4.3 monitoring dump
We don't want to delete PV after StatefulSet downscaling as this might lead to data loss. Essentially we want to have a healthy setup even after user does something that is not supported (manually scaling SS), for this we can ensure number of replicas in StatefulSet stays equal to what was specified in Prometheus CR. From what I see that is exactly what happened and that is expected. In short: manually scaling prometheus or alertmanager StatefulSet is not supported and might lead to having some artifacts left behind. However, it should not affect a cluster and number of pods should be the same before and after manual scaling. > We scaled up statefulset prometheus-k8s to 3, not scale down.
Yes, but then operator needs to react on that and immediately scale it down to a number of replicas specified in Prometheus CR (2 in this case), which is exactly what happens.
Closing as NOTABUG.
|
Created attachment 1614388 [details] ClusterMonitoringOperatorErrors alert