This is not a bug. The monitoring stack introduced in 3.11 is a managed stack, it is automatically updated and therefore exposes a minimal amount of customization, in order for us to be able to ensure sane upgrade paths.

You mean that you will not allow grafana to specify storage settings during the installation, but we need to change it after the installation manually?
Many users expect one operation for the installation, though.

Neither, the stack is shipped and configured as is. Any configuration parameter is a feature request, but adding storage to this Grafana should not be necessary as it's automatically provisioned at startup with all the information it needs. It hints that the customer wants to do something that the stack is not intended to be used for (monitor exactly the components that are pre-configured with the pre-configured dashboards, more is a feature request). For example we do have feature requests for the ability to add additional dashboards, but that doesn't mean the Grafana server must be configured to have storage.

OK, let me rephrase the question. When you installed cluster with grafana, what volume (PV/PVC) is attached to "/var/lib/grafana/" in your grafana pod? We observed emptyDir is always used for "/var/lib/grafana/" and cannot change it.

Until 3.10, we could configure the persistent volume for "/var/lib/grafana/" by using "openshift_grafana_storage_*" parameters.

We always use an emptyDir because Grafana is statically provisioned from configuration files.

The Grafana role still, exists so you can still create new Grafana servers outside of the cluster monitoring stack should you want to be able to create additional dashboards.

It looks like you assumed that users use only the default dashboard, which the grafana image provides by default. Some users want to customize dashboard or import plugins. These info added by users will be gone without persistent volumes.

Hi Kenjiro.

The decision to make the Grafana, which is being shipped with the new cluster monitoring stack, not editable was a very conscious decision that actually spans across all components of the monitoring stack. Let me highlight the core why's:

- Any arbitrary configurations could break the entire monitoring stack during, for example, installation, not just a single component.
- Our vision for an automated over-the-air update for OpenShift very much depends on how much we know about a system. As soon as we give away all control, customers will completely loose over-the-air updates. That could have impact on their ability to receive critical security patches, for example. Obviously, there are more reasons why over-the-air updates is critical for Red Hat's next stage.

Nevertheless, we understand the demand and we have that on our roadmap. Please understand that we need to do some investigations first to ensure we don't mess around with the points above.

Quick question. How did you use Grafana before? I neither see any supported option nor documentation for that in previous versions. Where did you connect Grafana to to get data?

Hi Christian,

> - Our vision for an automated over-the-air update for OpenShift very much depends on how much we know about a system. As soon as we give away all control, customers will completely loose over-the-air updates. That could have impact on their ability to receive critical security patches, for example. Obviously, there are more reasons why over-the-air updates is critical for Red Hat's next stage.

I understand and it makes sense. But it is very controversial especially above point. Actually the customer who reported this request (=add openshift_grafana_storage_* parameters), they did workaround by manually adding PV to the grafana by themselves after the installation. Do you mean that it may cause a problem when they will receive patches?

> Quick question. How did you use Grafana before? I neither see any supported option nor documentation for that in previous versions. Where did you connect Grafana to to get data?

Sure, you can refer to following docs:

  OpenShift Grafana Playbooks
  https://github.com/openshift/openshift-ansible/tree/release-3.10/roles/openshift_grafana

It is shipped via the package:

  # rpm -qf /usr/share/ansible/openshift-ansible/roles/openshift_grafana/
  openshift-ansible-roles-3.10.41-1.git.0.fd15dd7.el7.noarch

(We know 3.10's grafana was not supported.)

> Do you mean that it may cause a problem when they will receive patches?

You are saying that they configured the Grafana instance inside the openshift-monitoring namespace?

> You are saying that they configured the Grafana instance inside the openshift-monitoring namespace?

That is good question. I have not confirmed it with the customer, but I just heard that they changed it manually. I guess that if we changed Grafana in openshift-monitoring namespace "somehow", it may cause some problem. While if it was not managed by prometheus-operator then it should be alright. Correct?

I asked since the cluster monitoring operator refreshes the state on the deployed objects every x minutes to it's expected state. Means, they can change but the change will be gone with the next refresh cycle. :)

I am currently confirming why the Grafana playbook/role has been removed. Although it's an unsupported role, maybe its still enough for you ands your customer until we figured out how we enable these things without screwing around with the update process.

Let's close this issue as won't fix since it is not intended to extend/modify the cluster-monitoring stack including Grafana. We have an RFE open to provide a more flexible Grafana environment in the future. Please see here: https://bugzilla.redhat.com/show_bug.cgi?id=1668441