Created attachment 1791781 [details]
screenshot #1: ocs dashboard with list of firing alerts, see KubePersistentVolumeFillingUp there

Created attachment 1794347 [details]
screenshot of gp2 PVC with 0% free after creation

This is not only a false alert for LSO PVCs, also for PVCs created on gp2. Maybe the subject of this BZ can be adjusted?

And PVC with "volumeMode: Block" would be affected with this.

Niels/Anmol, what are the next steps here? OCP 4.8 is already in Freeze stage and if this is a blocker, we might have to target this for OCP 4.8.1 so that it can be shipped before OCS4.8 releases.

I think the approach should be to adjust the alerting rules. These alerts (KubePersistentVolumeFillingUp) should not get fired for `volumeMode: Block` volumes as usage/free can not be detected for them (both are set to 0, only capacity is valid).

reproduced with comments #c15

Changes from upstream kubernetes-mixin to CMO will be synced through https://github.com/openshift/cluster-monitoring-operator/pull/1269

Test with payload 4.9.0-0.nightly-2021-07-15-015134

Following #c15, no alerts KubePersistentVolumeFillingUp is triggered

Check alert rule changed
#oc -n openshift-monitoring get cm prometheus-k8s-rulefiles-0 -oyaml|grep KubePersistentVolumeFillingUp -A20
      - alert: KubePersistentVolumeFillingUp
        annotations:
          description: The PersistentVolume claimed by {{ $labels.persistentvolumeclaim
            }} in Namespace {{ $labels.namespace }} is only {{ $value | humanizePercentage
            }} free.
          summary: PersistentVolume is filling up.
        expr: |
          (
            kubelet_volume_stats_available_bytes{namespace=~"(openshift-.*|kube-.*|default)",job="kubelet", metrics_path="/metrics"}
              /
            kubelet_volume_stats_capacity_bytes{namespace=~"(openshift-.*|kube-.*|default)",job="kubelet", metrics_path="/metrics"}
          ) < 0.03
          and
          kubelet_volume_stats_used_bytes{namespace=~"(openshift-.*|kube-.*|default)",job="kubelet", metrics_path="/metrics"} > 0
        for: 1m
        labels:
          severity: critical
      - alert: KubePersistentVolumeFillingUp
        annotations:
          description: Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim
            }} in Namespace {{ $labels.namespace }} is expected to fill up within four
            days. Currently {{ $value | humanizePercentage }} is available.
          summary: PersistentVolume is filling up.
        expr: |
          (
            kubelet_volume_stats_available_bytes{namespace=~"(openshift-.*|kube-.*|default)",job="kubelet", metrics_path="/metrics"}
              /
            kubelet_volume_stats_capacity_bytes{namespace=~"(openshift-.*|kube-.*|default)",job="kubelet", metrics_path="/metrics"}
          ) < 0.15
          and
          kubelet_volume_stats_used_bytes{namespace=~"(openshift-.*|kube-.*|default)",job="kubelet", metrics_path="/metrics"} > 0
          and
          predict_linear(kubelet_volume_stats_available_bytes{namespace=~"(openshift-.*|kube-.*|default)",job="kubelet", metrics_path="/metrics"}[6h], 4 * 24 * 3600) < 0
        for: 1h
        labels:
          severity: warning

*** Bug 1986917 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days