Created attachment 1614388 [details] ClusterMonitoringOperatorErrors alert
Created attachment 1614389 [details] monitoring dump
Created attachment 1626292 [details] 4.3 monitoring dump
We don't want to delete PV after StatefulSet downscaling as this might lead to data loss. Essentially we want to have a healthy setup even after user does something that is not supported (manually scaling SS), for this we can ensure number of replicas in StatefulSet stays equal to what was specified in Prometheus CR. From what I see that is exactly what happened and that is expected. In short: manually scaling prometheus or alertmanager StatefulSet is not supported and might lead to having some artifacts left behind. However, it should not affect a cluster and number of pods should be the same before and after manual scaling.
> We scaled up statefulset prometheus-k8s to 3, not scale down. Yes, but then operator needs to react on that and immediately scale it down to a number of replicas specified in Prometheus CR (2 in this case), which is exactly what happens. Closing as NOTABUG.