Description of problem:
secret grpc-tls is missing in a fresh cluster(techPreviewUserWorkload is disabled), it causes the monitoring DEGRADED.
workaround is: enable techPreviewUserWorkload, secret grpc-tls is created, monitoring will be fine
# oc get co/monitoring
NAME         VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
monitoring   4.5.0-0.nightly-2020-04-23-202137   False       True          True       10m

# oc get co/monitoring -oyaml
...
  - lastTransitionTime: "2020-04-24T02:52:07Z"
    message: 'Failed to rollout the stack. Error: running task Updating Prometheus-k8s
      failed: waiting for Prometheus GRPC secret failed: waiting for secret grpc-tls:
      secrets "grpc-tls" not found'
    reason: UpdatingPrometheusK8SFailed
    status: "True"
    type: Degraded
...

# oc -n openshift-monitoring get secret grpc-tls
Error from server (NotFound): secrets "grpc-tls" not found

# oc -n openshift-monitoring logs thanos-querier-7fbf9ff6c-6x7lv -c thanos-query
level=info ts=2020-04-24T03:10:47.106024552Z caller=main.go:152 msg="Tracing will be disabled"
level=info ts=2020-04-24T03:10:47.106143629Z caller=client.go:54 msg="enabling client to server TLS"
level=info ts=2020-04-24T03:10:47.106341469Z caller=options.go:76 msg="TLS client using provided certificate pool"
level=info ts=2020-04-24T03:10:47.10656252Z caller=options.go:104 msg="TLS client authentication enabled"
level=info ts=2020-04-24T03:10:47.107998766Z caller=options.go:23 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2020-04-24T03:10:47.108459858Z caller=query.go:401 msg="starting query node"
level=info ts=2020-04-24T03:10:47.11219967Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=info ts=2020-04-24T03:10:47.112400222Z caller=grpc.go:106 service=gRPC/server component=query msg="listening for StoreAPI gRPC" address=127.0.0.1:10901
level=info ts=2020-04-24T03:10:47.112469774Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
level=info ts=2020-04-24T03:10:47.112485375Z caller=http.go:56 service=http/server component=query msg="listening for requests and metrics" address=127.0.0.1:9090
level=warn ts=2020-04-24T03:10:52.119682521Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.129.2.15:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.129.2.15:10901
level=warn ts=2020-04-24T03:10:52.119961596Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.131.0.18:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.131.0.18:10901
level=warn ts=2020-04-24T03:10:57.1203892Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.129.2.15:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.129.2.15:10901
level=warn ts=2020-04-24T03:10:57.120423833Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.131.0.18:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.131.0.18:10901
level=warn ts=2020-04-24T03:11:02.12090855Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.131.0.18:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.131.0.18:10901
level=warn ts=2020-04-24T03:11:02.120979767Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.129.2.15:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.129.2.15:10901
level=warn ts=2020-04-24T03:11:03.707892697Z caller=proxy.go:291 err="No StoreAPIs matched for this query" stores=
level=warn ts=2020-04-24T03:11:03.87547059Z caller=proxy.go:291 err="No StoreAPIs matched for this query" stores=
level=warn ts=2020-04-24T03:11:03.875608288Z caller=proxy.go:291 err="No StoreAPIs matched for this query" stores=
level=warn ts=2020-04-24T03:11:07.121372416Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.131.0.18:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.131.0.18:10901
level=warn ts=2020-04-24T03:11:07.121372534Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.129.2.15:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.129.2.15:10901
level=warn ts=2020-04-24T03:11:12.121743177Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.131.0.18:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.131.0.18:10901
level=warn ts=2020-04-24T03:11:12.121747416Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.129.2.15:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.129.2.15:10901
level=warn ts=2020-04-24T03:11:17.122143971Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.131.0.18:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.131.0.18:10901
level=warn ts=2020-04-24T03:11:17.122143216Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.129.2.15:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.129.2.15:10901
level=warn ts=2020-04-24T03:11:22.122539473Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.131.0.18:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.131.0.18:10901
level=warn ts=2020-04-24T03:11:22.122595043Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.129.2.15:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.129.2.15:10901
level=warn ts=2020-04-24T03:11:27.12300226Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.129.2.15:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.129.2.15:10901
level=warn ts=2020-04-24T03:11:27.122992631Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.131.0.18:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.131.0.18:10901
level=warn ts=2020-04-24T03:11:32.123431404Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.131.0.18:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.131.0.18:10901
level=warn ts=2020-04-24T03:11:32.123446129Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.129.2.15:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.129.2.15:10901
level=warn ts=2020-04-24T03:11:37.123824208Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.131.0.18:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.131.0.18:10901
level=warn ts=2020-04-24T03:11:37.123820601Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.129.2.15:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.129.2.15:10901
level=warn ts=2020-04-24T03:11:42.124246994Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 10.129.2.15:10901: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\"" address=10.129.2.15:10901

if we enable techPreviewUserWorkload, monitoring will be fine
# oc -n openshift-monitoring get secret grpc-tls
NAME       TYPE     DATA   AGE
grpc-tls   Opaque   6      11m

# oc get co/monitoring
NAME         VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
monitoring   4.5.0-0.nightly-2020-04-23-202137   True        False         False      16m


Version-Release number of selected component (if applicable):
4.5.0-0.nightly-2020-04-23-202137

How reproducible:
Always

Steps to Reproduce:
1. In a fresh cluster
2.
3.

Actual results:
monitoring DEGRADED

Expected results:
no error

Additional info: