Bug 1932229
| Summary: | Can’t find kubelet metrics for aws ebs csi volumes | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Qin Ping <piqin> |
| Component: | Storage | Assignee: | Jan Safranek <jsafrane> |
| Storage sub component: | Kubernetes External Components | QA Contact: | Qin Ping <piqin> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | aos-bugs, jsafrane |
| Version: | 4.7 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-27 22:48:06 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
GET_VOLUME_STATS capability was added recently in https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/677. We shall get it with the next rebase. Verified with: 4.8.0-0.nightly-2021-03-06-055252 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |
Description of problem: Can’t find kubelet metrics for aws ebs csi volumes Version-Release number of selected component (if applicable): 4.7.0-rc.3 How reproducible: Always Steps to Reproduce: Have an 4.7 OCP cluster on AWS Create a PVC(test-pvc-7) with gp2-csi storageclass Create a Pod using this PVC Create a PVC(test-pvc-8) with gp2 storageclass Create a Pod using this PVC When both pods are running check the kubelet metrics Actual results: $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE test-pvc-7 Bound pvc-895d4d67-9ae9-4225-a5fa-9d0e725962e8 4Gi RWO gp2-csi 21m test-pvc-8 Bound pvc-6bc18ca8-7bb3-495c-8006-4e94b367561f 4Gi RWO gp2 19m $ oc get deployment deployment-7 -o json|jq .spec.template.spec.volumes [ { "name": "local", "persistentVolumeClaim": { "claimName": "test-pvc-7" } } ] $ oc get deployment deployment-8 -o json|jq .spec.template.spec.volumes [ { "name": "local", "persistentVolumeClaim": { "claimName": "test-pvc-8" } } ] $ oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES deployment-7-7d96c76884-m8cvw 1/1 Running 0 14m 10.128.0.82 ip-10-0-187-153.us-east-2.compute.internal <none> <none> deployment-8-64f9679685-cb6rx 1/1 Running 0 12m 10.128.0.83 ip-10-0-187-153.us-east-2.compute.internal <none> <none> $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://10.0.187.153:10250/metrics' | grep kubelet_volume % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0# HELP kubelet_volume_stats_available_bytes [ALPHA] Number of available bytes in the volume # TYPE kubelet_volume_stats_available_bytes gauge kubelet_volume_stats_available_bytes{namespace="default",persistentvolumeclaim="test-pvc-8"} 4.126875648e+09 # HELP kubelet_volume_stats_capacity_bytes [ALPHA] Capacity in bytes of the volume # TYPE kubelet_volume_stats_capacity_bytes gauge kubelet_volume_stats_capacity_bytes{namespace="default",persistentvolumeclaim="test-pvc-8"} 4.160421888e+09 # HELP kubelet_volume_stats_inodes [ALPHA] Maximum number of inodes in the volume # TYPE kubelet_volume_stats_inodes gauge kubelet_volume_stats_inodes{namespace="default",persistentvolumeclaim="test-pvc-8"} 262144 # HELP kubelet_volume_stats_inodes_free [ALPHA] Number of free inodes in the volume # TYPE kubelet_volume_stats_inodes_free gauge kubelet_volume_stats_inodes_free{namespace="default",persistentvolumeclaim="test-pvc-8"} 262133 # HELP kubelet_volume_stats_inodes_used [ALPHA] Number of used inodes in the volume # TYPE kubelet_volume_stats_inodes_used gauge kubelet_volume_stats_inodes_used{namespace="default",persistentvolumeclaim="test-pvc-8"} 11 # HELP kubelet_volume_stats_used_bytes [ALPHA] Number of used bytes in the volume # TYPE kubelet_volume_stats_used_bytes gauge kubelet_volume_stats_used_bytes{namespace="default",persistentvolumeclaim="test-pvc-8"} 1.6769024e+07 100 251k 0 251k 0 0 15.3M 0 --:--:-- --:--:-- --:--:-- 15.3M Only can get kubelet metrics for test-pvc-8(the volume created by the internal provisioner). Expected results: Can get kubelet metrics for test-pvc-7 Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info: