Description of problem: Cloned from https://jira.coreos.com/browse/MON-587 "x509: certificate is valid for localhost" for openshift-controller-manager-operator and kube-controller-manager-operator targets, but could get the metrics from api, it just shows error in /targets, see below "discoveredLabels": { "__address__": "10.128.0.22:8443", "__meta_kubernetes_endpoint_address_target_kind": "Pod", "__meta_kubernetes_endpoint_address_target_name": "openshift-controller-manager-operator-76f66d9b65-45vgn", "__meta_kubernetes_endpoint_port_name": "https", "__meta_kubernetes_endpoint_port_protocol": "TCP", "__meta_kubernetes_endpoint_ready": "true", "__meta_kubernetes_endpoints_name": "metrics", "__meta_kubernetes_namespace": "openshift-controller-manager-operator", "__meta_kubernetes_pod_annotation_k8s_v1_cni_cncf_io_networks_status": "[{\n \"name\": \"openshift-sdn\",\n \"ips\": [\n \"10.128.0.22\"\n ],\n \"default\": true,\n \"dns\": {}\n}]", "__meta_kubernetes_pod_container_name": "operator", "__meta_kubernetes_pod_container_port_name": "metrics", "__meta_kubernetes_pod_container_port_number": "8443", "__meta_kubernetes_pod_container_port_protocol": "TCP", "__meta_kubernetes_pod_controller_kind": "ReplicaSet", "__meta_kubernetes_pod_controller_name": "openshift-controller-manager-operator-76f66d9b65", "__meta_kubernetes_pod_host_ip": "10.0.140.52", "__meta_kubernetes_pod_ip": "10.128.0.22", "__meta_kubernetes_pod_label_app": "openshift-controller-manager-operator", "__meta_kubernetes_pod_label_pod_template_hash": "76f66d9b65", "__meta_kubernetes_pod_name": "openshift-controller-manager-operator-76f66d9b65-45vgn", "__meta_kubernetes_pod_node_name": "ip-10-0-140-52.us-east-2.compute.internal", "__meta_kubernetes_pod_phase": "Running", "__meta_kubernetes_pod_ready": "true", "__meta_kubernetes_pod_uid": "3c501b7a-365c-11e9-9033-0287d15921aa", "__meta_kubernetes_service_annotation_service_alpha_openshift_io_serving_cert_secret_name": "openshift-controller-manager-operator-serving-cert", "__meta_kubernetes_service_annotation_service_alpha_openshift_io_serving_cert_signed_by": "openshift-service-serving-signer@1550809822", "__meta_kubernetes_service_label_app": "openshift-controller-manager-operator", "__meta_kubernetes_service_name": "metrics", "__metrics_path__": "/metrics", "__scheme__": "https", "job": "openshift-controller-manager-operator/openshift-controller-manager-operator/0" }, "health": "down", "labels": { "endpoint": "https", "instance": "10.128.0.22:8443", "job": "metrics", "namespace": "openshift-controller-manager-operator", "pod": "openshift-controller-manager-operator-76f66d9b65-45vgn", "service": "metrics" }, "lastError": "Get https://10.128.0.22:8443/metrics: x509: certificate is valid for localhost, not metrics.openshift-controller-manager-operator.svc", "lastScrape": "2019-02-22T07:52:51.626332327Z", "scrapeUrl": "https://10.128.0.22:8443/metrics" **************************************************************************************************************** "discoveredLabels": { "__address__": "10.128.0.6:8443", "__meta_kubernetes_endpoint_address_target_kind": "Pod", "__meta_kubernetes_endpoint_address_target_name": "kube-controller-manager-operator-54f59bbf4f-29dnr", "__meta_kubernetes_endpoint_port_name": "https", "__meta_kubernetes_endpoint_port_protocol": "TCP", "__meta_kubernetes_endpoint_ready": "true", "__meta_kubernetes_endpoints_name": "metrics", "__meta_kubernetes_namespace": "openshift-kube-controller-manager-operator", "__meta_kubernetes_pod_annotation_k8s_v1_cni_cncf_io_networks_status": "[{\n \"name\": \"openshift-sdn\",\n \"ips\": [\n \"10.128.0.6\"\n ],\n \"default\": true,\n \"dns\": {}\n}]", "__meta_kubernetes_pod_container_name": "operator", "__meta_kubernetes_pod_container_port_name": "metrics", "__meta_kubernetes_pod_container_port_number": "8443", "__meta_kubernetes_pod_container_port_protocol": "TCP", "__meta_kubernetes_pod_controller_kind": "ReplicaSet", "__meta_kubernetes_pod_controller_name": "kube-controller-manager-operator-54f59bbf4f", "__meta_kubernetes_pod_host_ip": "10.0.140.52", "__meta_kubernetes_pod_ip": "10.128.0.6", "__meta_kubernetes_pod_label_app": "kube-controller-manager-operator", "__meta_kubernetes_pod_label_pod_template_hash": "54f59bbf4f", "__meta_kubernetes_pod_name": "kube-controller-manager-operator-54f59bbf4f-29dnr", "__meta_kubernetes_pod_node_name": "ip-10-0-140-52.us-east-2.compute.internal", "__meta_kubernetes_pod_phase": "Running", "__meta_kubernetes_pod_ready": "true", "__meta_kubernetes_pod_uid": "4f98f87b-365b-11e9-8b9a-0289e5c3d940", "__meta_kubernetes_service_annotation_service_alpha_openshift_io_serving_cert_secret_name": "kube-controller-manager-operator-serving-cert", "__meta_kubernetes_service_annotation_service_alpha_openshift_io_serving_cert_signed_by": "openshift-service-serving-signer@1550809822", "__meta_kubernetes_service_label_app": "kube-controller-manager-operator", "__meta_kubernetes_service_name": "metrics", "__metrics_path__": "/metrics", "__scheme__": "https", "job": "openshift-kube-controller-manager-operator/kube-controller-manager-operator/0" }, "health": "down", "labels": { "endpoint": "https", "instance": "10.128.0.6:8443", "job": "metrics", "namespace": "openshift-kube-controller-manager-operator", "pod": "kube-controller-manager-operator-54f59bbf4f-29dnr", "service": "metrics" }, "lastError": "Get https://10.128.0.6:8443/metrics: x509: certificate is valid for localhost, not metrics.openshift-kube-controller-manager-operator.svc", "lastScrape": "2019-02-22T07:52:46.648616909Z", "scrapeUrl": "https://10.128.0.6:8443/metrics" $ curl -k -H "Authorization: Bearer $(oc sa get-token prometheus-k8s -n openshift-monitoring)" https://10.128.0.22:8443/metrics # HELP ConfigObserver_adds Total number of adds handled by workqueue: ConfigObserver # TYPE ConfigObserver_adds counter ConfigObserver_adds 721 # HELP ConfigObserver_depth Current depth of workqueue: ConfigObserver # TYPE ConfigObserver_depth gauge ConfigObserver_depth 1 # HELP ConfigObserver_queue_latency How long an item stays in workqueueConfigObserver before being requested. # TYPE ConfigObserver_queue_latency summary ConfigObserver_queue_latency{quantile="0.5"} 1.8688621e+07 ConfigObserver_queue_latency{quantile="0.9"} 1.9818535e+07 ConfigObserver_queue_latency{quantile="0.99"} 1.9923432e+07 ConfigObserver_queue_latency_sum 1.3578328876e+10 ConfigObserver_queue_latency_count 720 # HELP ConfigObserver_retries Total number of retries handled by workqueue: ConfigObserver # TYPE ConfigObserver_retries counter ConfigObserver_retries 0 # HELP ConfigObserver_work_duration How long processing an item from workqueueConfigObserver takes. # TYPE ConfigObserver_work_duration summary $ curl -k -H "Authorization: Bearer $(oc sa get-token prometheus-k8s -n openshift-monitoring)" https://10.128.0.6:8443/metrics # HELP BackingResourceController_adds Total number of adds handled by workqueue: BackingResourceController # TYPE BackingResourceController_adds counter BackingResourceController_adds 391 # HELP BackingResourceController_depth Current depth of workqueue: BackingResourceController # TYPE BackingResourceController_depth gauge BackingResourceController_depth 0 # HELP BackingResourceController_queue_latency How long an item stays in workqueueBackingResourceController before being requested. # TYPE BackingResourceController_queue_latency summary BackingResourceController_queue_latency{quantile="0.5"} 4 BackingResourceController_queue_latency{quantile="0.9"} 335286 BackingResourceController_queue_latency{quantile="0.99"} 335286 BackingResourceController_queue_latency_sum 4.2015996e+07 BackingResourceController_queue_latency_count 391 # HELP BackingResourceController_retries Total number of retries handled by workqueue: BackingResourceController # TYPE BackingResourceController_retries counter BackingResourceController_retries 0 # HELP BackingResourceController_work_duration How long processing an item from workqueueBackingResourceController takes. # TYPE BackingResourceController_work_duration summary BackingResourceController_work_duration{quantile="0.5"} 187615 BackingResourceController_work_duration{quantile="0.9"} 790087 Version-Release number of selected component (if applicable): $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-02-21-215247 True False 4h4m Cluster version is 4.0.0-0.nightly-2019-02-21-215247 How reproducible: Frequently Steps to Reproduce: 1. Check all targets via REST API curl -k -H "Authorization: Bearer $(oc sa get-token prometheus-k8s -n openshift-monitoring)" https://{prometheus_ip}:9091/api/v1/targets | python -mjson.tool 2. 3. Actual results: Expected results: Additional info:
This is not a bug in monitoring, this is the openshift-controller-manager-operator and kube-controller-manager-operator targets not presenting the expected certificates. Moving to master component.
*** This bug has been marked as a duplicate of bug 1678929 ***