Bug 1840820

Summary: Additional PVCs created in openshift-monitoring namespace after upgrading from 4.3 to 4.4.4
Product: OpenShift Container Platform Reporter: Aditya Deshpande <adeshpan>
Component: MonitoringAssignee: Pawel Krupa <pkrupa>
Status: CLOSED DUPLICATE QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.4CC: aghadge, alegrand, anpicker, erooth, kakkoyun, lcosic, mloibl, pkrupa, surbania
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-28 09:52:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Aditya Deshpande 2020-05-27 16:36:23 UTC
Description of problem:

After upgrading from OCP 4.3.21 to 4.4.4, there are additional PVCs getting created for prometheus and alertmanager though there were PVCs present before upgrade.

Before upgrade: 
~~~
# oc get pvc
NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
prom-prometheus-k8s-0                Bound    pvc-af243bba-8281-4617-965b-a9cf43adcf41   1Gi        RWO            gp2            4h59m
prom-prometheus-k8s-1                Bound    pvc-679cf5a9-83d7-4927-b15f-fbfecf824720   1Gi        RWO            gp2            4h59m

# oc get cm cluster-monitoring-config -o yaml
apiVersion: v1
data:
  config.yaml: |
    prometheusK8s:
      volumeClaimTemplate:
        metadata:
          name: prom          <==== PVC-prefix-name
        spec:
          storageClassName: gp2
          resources:
            requests:
              storage: 1Gi
kind: ConfigMap
~~~

Here, it was already PVCs were present with PVC-prefix-name: "prom"

But after  upgrade to 4.4.4, it is observed that additional PVC with default PVC-prefix-name got created and new PVCs are getting used by the respective pods.

After upgrade:
~~~
# oc get pvc
NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
prom-prometheus-k8s-0                Bound    pvc-af243bba-8281-4617-965b-a9cf43adcf41   1Gi        RWO            gp2            4h59m
prom-prometheus-k8s-1                Bound    pvc-679cf5a9-83d7-4927-b15f-fbfecf824720   1Gi        RWO            gp2            4h59m
prometheus-k8s-db-prometheus-k8s-0   Bound    pvc-ef25921b-1a59-4145-ad9f-14904f0feb97   1Gi        RWO            gp2            109m
prometheus-k8s-db-prometheus-k8s-1   Bound    pvc-6e8ea9fd-eb97-4a3a-a1ef-32319b615fa1   1Gi        RWO            gp2            109m

# oc describe pod prometheus-k8s-0
Volumes:
  prometheus-k8s-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  prometheus-k8s-db-prometheus-k8s-0
    ReadOnly:   false
~~~

Version-Release number of selected component (if applicable):
# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.4     True        False         69m     Cluster version is 4.4.4

How reproducible:
Always

Steps to Reproduce:
1. deploy OCP v4.3.21 with customized PVC-prefix-name for prometheus and alertmanager pods with persistent storage.
2. upgrade to 4.4.4 using stable channel
3. check the pvc inside openshift-monitoring namespace after upgrade

Actual results:
Additional PVCs with default pvc-prefix-name are getting created though there are already present PVCs which are created before upgrade. Existing PVCs are not being used by the new prometheus and alertmanager pods.

Expected results:
Prometheus and alertmanager pods should use the existing PVCs which are created before upgrade.
Also, there should not be any new creation of PVCs with default pvc-prefix-name as there are PVCs present already.

Additional info:
The cluster-monitoring-config configmpa inside the openshift-monitoring namespace is not changed during upgrade process.

Comment 2 Pawel Krupa 2020-05-28 09:52:26 UTC

*** This bug has been marked as a duplicate of bug 1833427 ***