Description of problem:
Can’t find kubelet metrics for aws ebs csi volumes

Version-Release number of selected component (if applicable):
4.7.0-rc.3

How reproducible:
Always

Steps to Reproduce:
Have an 4.7 OCP cluster on AWS
Create a PVC(test-pvc-7) with gp2-csi storageclass
Create a Pod using this PVC
Create a PVC(test-pvc-8) with gp2 storageclass
Create a Pod using this PVC
When both pods are running check the kubelet metrics

Actual results:

$ oc get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-pvc-7   Bound    pvc-895d4d67-9ae9-4225-a5fa-9d0e725962e8   4Gi        RWO            gp2-csi        21m
test-pvc-8   Bound    pvc-6bc18ca8-7bb3-495c-8006-4e94b367561f   4Gi        RWO            gp2            19m

$ oc get deployment deployment-7 -o json|jq .spec.template.spec.volumes
[
  {
    "name": "local",
    "persistentVolumeClaim": {
      "claimName": "test-pvc-7"
    }
  }
]

$ oc get deployment deployment-8 -o json|jq .spec.template.spec.volumes
[
  {
    "name": "local",
    "persistentVolumeClaim": {
      "claimName": "test-pvc-8"
    }
  }
]


$ oc get pod -o wide
NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE                                         NOMINATED NODE   READINESS GATES
deployment-7-7d96c76884-m8cvw   1/1     Running   0          14m   10.128.0.82   ip-10-0-187-153.us-east-2.compute.internal   <none>           <none>
deployment-8-64f9679685-cb6rx   1/1     Running   0          12m   10.128.0.83   ip-10-0-187-153.us-east-2.compute.internal   <none>           <none>


$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://10.0.187.153:10250/metrics' | grep kubelet_volume
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP kubelet_volume_stats_available_bytes [ALPHA] Number of available bytes in the volume
# TYPE kubelet_volume_stats_available_bytes gauge
kubelet_volume_stats_available_bytes{namespace="default",persistentvolumeclaim="test-pvc-8"} 4.126875648e+09
# HELP kubelet_volume_stats_capacity_bytes [ALPHA] Capacity in bytes of the volume
# TYPE kubelet_volume_stats_capacity_bytes gauge
kubelet_volume_stats_capacity_bytes{namespace="default",persistentvolumeclaim="test-pvc-8"} 4.160421888e+09
# HELP kubelet_volume_stats_inodes [ALPHA] Maximum number of inodes in the volume
# TYPE kubelet_volume_stats_inodes gauge
kubelet_volume_stats_inodes{namespace="default",persistentvolumeclaim="test-pvc-8"} 262144
# HELP kubelet_volume_stats_inodes_free [ALPHA] Number of free inodes in the volume
# TYPE kubelet_volume_stats_inodes_free gauge
kubelet_volume_stats_inodes_free{namespace="default",persistentvolumeclaim="test-pvc-8"} 262133
# HELP kubelet_volume_stats_inodes_used [ALPHA] Number of used inodes in the volume
# TYPE kubelet_volume_stats_inodes_used gauge
kubelet_volume_stats_inodes_used{namespace="default",persistentvolumeclaim="test-pvc-8"} 11
# HELP kubelet_volume_stats_used_bytes [ALPHA] Number of used bytes in the volume
# TYPE kubelet_volume_stats_used_bytes gauge
kubelet_volume_stats_used_bytes{namespace="default",persistentvolumeclaim="test-pvc-8"} 1.6769024e+07
100  251k    0  251k    0     0  15.3M      0 --:--:-- --:--:-- --:--:-- 15.3M

Only can get kubelet metrics for test-pvc-8(the volume created by the internal provisioner).

Expected results:
Can get kubelet metrics for test-pvc-7

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info: