Bug 1918599

Summary: Persistent storage documentation for Monitoring stack needs to be reviewed
Product: OpenShift Container Platform Reporter: Nishant Chauhan <nishantchauhan>
Component: DocumentationAssignee: Claire Bremble <cbremble>
Status: CLOSED CURRENTRELEASE QA Contact: Xiaoli Tian <xtian>
Severity: low Docs Contact: Claire Bremble <cbremble>
Priority: medium    
Version: 4.8CC: aos-bugs, cbremble, jfajersk, sniemann
Target Milestone: ---   
Target Release: ---   
Hardware: s390x   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-02-10 16:02:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1903544    

Description Nishant Chauhan 2021-01-21 07:52:37 UTC
I was not able to re-open the below bug, so raising a new one.
Refer BZ : https://bugzilla.redhat.com/show_bug.cgi?id=1788502


Document URL: 

https://docs.openshift.com/container-platform/4.6/monitoring/configuring-the-monitoring-stack.html#configuring-a-local-persistent-volume-claim_configuring-the-monitoring-stack

Section Number and Name: 

Unknown


Describe the issue: 

[root@bastion mon]# oc version
Client Version: 4.7.0-0.nightly-s390x-2020-12-21-160105
Server Version: 4.7.0-fc.2
Kubernetes Version: v1.20.0+394a5a3

As per the documentation

https://docs.openshift.com/container-platform/4.6/monitoring/configuring-the-monitoring-stack.html#configuring-a-local-persistent-volume-claim_configuring-the-monitoring-stack

Prerequisites

- Dedicate sufficient local persistent storage to ensure that the disk does not become full. How much storage you need depends on the number of pods. For information on system requirements for persistent storage, see Prometheus database storage requirements.

- Make sure you have a Persistent Volume (PV) ready to be claimed by the Persistent Volume Claim (PVC), one PV for each replica. Since Prometheus has two replicas and Alertmanager has three replicas, you need five PVs to support the entire monitoring stack. The Persistent Volumes should be available from the Local Storage Operator. This does not apply if you enable dynamically provisioned storage.

- Use the *block* type of storage.

- Configure local persistent storage

After configuring block storage type volume in LSO for Monitoring Stack , Pods are not coming up.  

---
[root@bastion mon]# oc get configmap cluster-monitoring-config -o yaml
apiVersion: v1
data:
  config.yaml: |
    prometheusK8s:
      volumeClaimTemplate:
        spec:
          accessModes:
          - ReadWriteOnce
          VolumeMode: Block
          storageClassName: localblock-new
          resources:
            requests:
              storage: 10Gi
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"config.yaml":"prometheusK8s:\n  volumeClaimTemplate:\n    spec:\n      accessModes:\n      - ReadWriteOnce\n      VolumeMode: Block\n      storageClassName: localblock-new\n      resources:\n        requests:\n          storage: 10Gi\n"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"cluster-monitoring-config","namespace":"openshift-monitoring"}}
  creationTimestamp: "2021-01-13T16:23:42Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:config.yaml: {}
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-01-13T16:23:42Z"
  name: cluster-monitoring-config
  namespace: openshift-monitoring
  resourceVersion: "3302593"
  selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config
  uid: b4024268-a9ff-4dda-b559-eb8d48dad847

---

---
[root@bastion mon]# oc get event -n openshift-monitoring --field-selector involvedObject.name=prometheus-k8s-0

  Warning  FailedMount       20s   kubelet            Unable to attach or mount volumes: unmounted volumes=[prometheus-k8s-db], unattached volumes=[secret-kube-rbac-proxy prometheus-trusted-ca-bundle secret-kube-etcd-client-certs prometheus-k8s-db tls-assets config config-out secret-prometheus-k8s-tls secret-prometheus-k8s-htpasswd configmap-kubelet-serving-ca-bundle prometheus-k8s-token-dzs6t configmap-serving-certs-ca-bundle secret-grpc-tls secret-prometheus-k8s-proxy prometheus-k8s-rulefiles-0 secret-prometheus-k8s-thanos-sidecar-tls]: volume prometheus-k8s-db has volumeMode Block, but is specified in volumeMounts
  Warning  FailedMount       8s    kubelet            Unable to attach or mount volumes: unmounted volumes=[prometheus-k8s-db], unattached volumes=[prometheus-trusted-ca-bundle prometheus-k8s-rulefiles-0 config secret-kube-etcd-client-certs secret-prometheus-k8s-thanos-sidecar-tls config-out configmap-serving-certs-ca-bundle configmap-kubelet-serving-ca-bundle tls-assets secret-prometheus-k8s-tls secret-prometheus-k8s-htpasswd prometheus-k8s-token-dzs6t secret-prometheus-k8s-proxy secret-grpc-tls prometheus-k8s-db secret-kube-rbac-proxy]: volume prometheus-k8s-db has volumeMode Block, but is specified in volumeMounts
----


Suggestions for improvement: 

May be replacing the *Block* mode prerequisite with *Filesystem* mode.

and according to previous bug ,https://bugzilla.redhat.com/show_bug.cgi?id=1788502 , it is not corrected in 4.2 also.


Additional information:

Comment 1 Silke Niemann 2021-01-21 16:37:52 UTC
@pkrupa In the previous bug you wrote "Changing requirement from *block* mode to *filesystem* mode sounds good to me." 

I can make the change in the docs but first need to know if this applies to all architectures or is this s390x specific?

Comment 2 Dan Li 2021-01-28 18:40:36 UTC
Hi Pavel - please see Silke's comment 1 regarding your input

Comment 11 Jan Fajerski 2022-07-20 13:05:52 UTC
(In reply to Silke Niemann from comment #1)
> @pkrupa In the previous bug you wrote "Changing requirement from *block*
> mode to *filesystem* mode sounds good to me." 
> 
> I can make the change in the docs but first need to know if this applies to
> all architectures or is this s390x specific?

Sorry about the delay, Pavel left the team in the meantime. Yes this applies to all architectures. Thanks!

Comment 14 Claire Bremble 2022-09-15 18:27:21 UTC
This impacts supported versions of OCP, so updating the version to 4.8

Comment 15 Claire Bremble 2023-02-10 16:02:27 UTC
This was closed via this Jira: https://issues.redhat.com/browse/RHDEVDOCS-4350