Bug 1485222
| Summary: | OSP metrics collected by Prometheus from the cAdvisor endpoint are missing labels | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Yaacov Zamir <yzamir> | ||||||
| Component: | Hawkular | Assignee: | Solly Ross <sross> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Junqi Zhao <juzhao> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 3.7.0 | CC: | aos-bugs, ccoleman, jcantril, miburman, mwringe, yzamir | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 3.7.0 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2017-10-31 15:53:26 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Created attachment 1318024 [details]
screenshot of cadvisor endpoint with labels
Have not verified whether core cadvisor is nroekn as wel, raising priority. Isn't this wrong place to track this? It's a development request, not a support request - 3.7 isn't a supported version. And this bugzilla isn't used to track new features. @jcantril: for Prometheus related issues, who should we assign them to? Seth, Paul Weil said this is something you could look into. I know of this fix for cadvisor wrt to Prometheus https://github.com/google/cadvisor/pull/1679 Is that related to this? I'm not well versed in Prometheus/metrics. If not, ask Solly (sross) I guess we should assign to Solly then. cAdvisor has no concept of containers in the same way Kubernetes has a concept of containers. To cAdvisor, a "container" is just a cgroup hierarchy. What you're seeing is cAdvisor's concept of containers. Since you're only seeing the containers with cAdivsor's concept of containers (and not those that also have Kube information attached), I'd guess you're hitting this bug: https://github.com/kubernetes/kubernetes/pull/51473. Does it alternate between working and not working? > Does it alternate between working and not working? a. the cAdvisor endpoint alternate ( e.g. https://hostname:10250/metrics/cadvisor) b. prometheus is missing the labels all the time. Clayton Coleman emailed that this was fixed in origin at the start of Sep, I did not checked the fix yet. Add Clayton for need info about the fix. Checked on origin 3.7 with k8s 1.7: # oc version oc v3.7.0-alpha.1+5250f65-1183 kubernetes v1.7.6+a08f5eeb62 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https:// :8443 openshift v3.7.0-alpha.1+5250f65-1183 kubernetes v1.7.6+a08f5eeb62 Works for me. |
Created attachment 1318023 [details] screenshot of cadvisor endpoint without labels Description of problem: OSP metrics collected by Prometheus from the cAdvisor endpoint are missing labels Version-Release number of selected component (if applicable): 3.7 ( 3.6 do not use the cAdvisor endpoint and is unaffected ) How reproducible: Consume OCP metrics ( e.g. use prometheus ui, grafana, manageiq ) Metrics originated from cAdvisor will miss labels, for example: "container_cpu_usage_seconds_total" will not have "container_name" Steps to Reproduce: 1. install OCP 3.7 2. install prometheus metrics engine (default metrics for 3.7) 3. consume OCP metrics Actual results: metrics only have an id label Expected results: metrics have id, name, namespace, project ... labels Additional info: attached screensots