Bug 1566495
| Summary: | CNS Prometheus metrics wrong | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Mangirdas Judeikis <mjudeiki> |
| Component: | Hawkular | Assignee: | Oved Ourfali <oourfali> |
| Status: | CLOSED NOTABUG | QA Contact: | Junqi Zhao <juzhao> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.9.0 | CC: | aos-bugs, jstrunk |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-04-12 15:42:41 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
So, when recreating it didnt deleted (done via ansible) underlying PV in the gluster. So [root@console-repl ~]# oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE registry-claim Bound registry-volume 10Gi RWX 4h sh-4.2$ df -h /registry/ Filesystem Size Used Avail Use% Mounted on 192.168.0.21:glusterfs-registry-volume 30G 30G 0 100% /registry So I think its ansible bug to clean storage, and not kubelet. |
1. Provision registry pv with 30G storage. 2. get metrics (kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="registry-claim"}) 3. Shows 30... profit.. 4. delete registry pvc, create new pvc with 10G storage. 5. Check pvc [root@console-REPL summit]# oc get pvc -n default NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE registry-claim Bound registry-volume 10Gi RWX 53m [root@console-REPL summit]# oc get pvc -n default NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE registry-claim Bound registry-volume 10Gi RWX 54m [root@console-REPL summit]# 6. check metrics: kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="registry-claim"} 7. still shows 30... Element Value kubelet_volume_stats_capacity_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",glusterfs="storage-host",instance="infra1.example.com",job="kubernetes-nodes",kubernetes_io_hostname="infra1.example.com",logging_infra_fluentd="true",namespace="default",persistentvolumeclaim="registry-claim",prometheus="true",region="infra",zone="az1"} 32192331776 kubelet_volume_stats_capacity_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",glusterfs="storage-host",instance="infra2.example.com",job="kubernetes-nodes",kubernetes_io_hostname="infra2.example.com",logging_infra_fluentd="true",namespace="default",persistentvolumeclaim="registry-claim",prometheus="true",region="infra",zone="az2"} 32192331776 kubelet_volume_stats_capacity_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",glusterfs="storage-host",instance="infra3.example.com",job="kubernetes-nodes",kubernetes_io_hostname="infra3.example.com",logging_infra_fluentd="true",namespace="default",persistentvolumeclaim="registry-claim",prometheus="true",region="infra",zone="az3"} 32192331776 8. rebounce nodes: ansible all -m shell -a "systemctl restart atomic-openshift-node" 9 same result: kubelet_volume_stats_capacity_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",glusterfs="storage-host",instance="infra2.example.com",job="kubernetes-nodes",kubernetes_io_hostname="infra2.example.com",logging_infra_fluentd="true",namespace="default",persistentvolumeclaim="registry-claim",prometheus="true",region="infra",zone="az2"}32192331776kubelet_volume_stats_capacity_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",glusterfs="storage-host",instance="infra3.example.com",job="kubernetes-nodes",kubernetes_io_hostname="infra3.example.com",logging_infra_fluentd="true",namespace="default",persistentvolumeclaim="registry-claim",prometheus="true",region="infra",zone="az3"} 32192331776