Bug 2089805
| Summary: | Config duration metrics aren't exposed | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Martin Kennelly <mkennell> |
| Component: | Networking | Assignee: | Martin Kennelly <mkennell> |
| Networking sub component: | ovn-kubernetes | QA Contact: | Weibin Liang <weliang> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | high | CC: | weliang |
| Version: | 4.11 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.11.0 | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Release Note | |
| Doc Text: |
OVN-Kubernetes CNI samples and exposes as prometheus metrics the time it takes to configure all nodes networking control plane with changes made to Kubernetes Kinds pods, services and network policy.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-08-10 11:14:04 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Tested and verified in 4.11.0-0.nightly-2022-06-06-201913
sh-4.4# curl 127.0.0.1:29102/metrics | grep ovnkube_master_network_programming_duration_seconds
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0# HELP ovnkube_master_network_programming_duration_seconds The duration to apply network configuration for a kind (e.g. pod, service, networkpolicy). Configuration includes add, update and delete events for each kind.
# TYPE ovnkube_master_network_programming_duration_seconds histogram
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="0"} 0
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="0.25"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="0.5"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="1"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="2"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="3"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="4"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="5"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="6"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="7"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="8"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="9"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="10"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="11"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="12"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="13"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="14"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="15"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="16"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="17"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="18"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="19"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="20"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="21"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="22"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="23"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="24"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="25"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="26"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="27"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="28"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="29"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="30"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="31"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="32"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="33"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="34"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="35"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="36"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="37"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="38"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="39"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="40"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="41"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="42"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="43"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="44"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="45"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="46"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="47"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="48"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="49"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="50"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="51"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="52"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="53"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="54"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="55"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="56"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="57"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="58"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="59"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="60"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="65"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="70"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="75"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="80"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="85"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="90"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="95"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="100"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="105"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="110"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="115"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="120"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="150"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="180"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="210"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="240"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="270"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="300"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="330"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="360"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="390"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="420"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="+Inf"} 60
ovnkube_master_network_programming_duration_seconds_sum{kind="pod"} 35.72184365000002
ovnkube_master_network_programming_duration_seconds_count{kind="pod"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="0"} 0
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="0.25"} 14
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="0.5"} 14
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="1"} 14
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="2"} 14
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="3"} 14
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="4"} 14
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="5"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="6"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="7"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="8"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="9"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="10"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="11"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="12"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="13"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="14"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="15"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="16"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="17"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="18"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="19"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="20"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="21"} 16
100 62414 0 62414 0 0 59.5M 0 --:--:-- --:--:-- --:--:-- 59.5M
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="22"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="23"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="24"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="25"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="26"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="27"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="28"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="29"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="30"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="31"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="32"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="33"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="34"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="35"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="36"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="37"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="38"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="39"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="40"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="41"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="42"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="43"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="44"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="45"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="46"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="47"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="48"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="49"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="50"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="51"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="52"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="53"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="54"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="55"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="56"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="57"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="58"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="59"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="60"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="65"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="70"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="75"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="80"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="85"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="90"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="95"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="100"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="105"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="110"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="115"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="120"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="150"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="180"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="210"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="240"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="270"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="300"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="330"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="360"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="390"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="420"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="+Inf"} 16
ovnkube_master_network_programming_duration_seconds_sum{kind="service"} 8.353540311000003
ovnkube_master_network_programming_duration_seconds_count{kind="service"} 16
sh-4.4#
sh-4.4# curl 127.0.0.1:29102/metrics | grep ovnkube_master_network_programming_ovn_duration_seconds
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0# HELP ovnkube_master_network_programming_ovn_duration_seconds The duration for OVN to apply network configuration
# TYPE ovnkube_master_network_programming_ovn_duration_seconds histogram
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="0"} 0
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="0.25"} 10
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="0.5"} 10
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="1"} 10
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="2"} 10
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="3"} 10
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="4"} 10
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="5"} 12
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069 |
Description of problem: Configuration duration monitoring is not enabled by default. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Curl the metrics endpoint for the ovnkube-master leader on port 9102. [1] 2. Search for 2 Prometheus bucket metrics: ovnkube_master_network_programming_ovn_duration_seconds_* and ovnkube_master_network_programming_duration_seconds_*. Actual results: Metrics not found. Expected results: Metrics found. Additional info: [1] Get the leader pod name from here: oc -n openshift-ovn-kubernetes get po -l app=ovnkube-master --field-selector spec.nodeName=$(oc -n openshift-ovn-kubernetes get cm ovn-kubernetes-master -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io\/leader}' | jq -r '.holderIdentity') -o jsonpath='{.items[0].metadata.name}'