Bug 2089805 - Config duration metrics aren't exposed
Summary: Config duration metrics aren't exposed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.11
Hardware: All
OS: All
high
medium
Target Milestone: ---
: 4.11.0
Assignee: Martin Kennelly
QA Contact: Weibin Liang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-24 13:32 UTC by Martin Kennelly
Modified: 2022-08-18 15:31 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Release Note
Doc Text:
OVN-Kubernetes CNI samples and exposes as prometheus metrics the time it takes to configure all nodes networking control plane with changes made to Kubernetes Kinds pods, services and network policy.
Clone Of:
Environment:
Last Closed: 2022-08-10 11:14:04 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-network-operator pull 1455 0 None Merged Bug 2089805: Enable config duration for OVN-Kubernetes 2022-06-13 15:38:25 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 11:14:15 UTC

Description Martin Kennelly 2022-05-24 13:32:02 UTC
Description of problem:
Configuration duration monitoring is not enabled by default.


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Curl the metrics endpoint for the ovnkube-master leader on port 9102. [1]
2. Search for 2 Prometheus bucket metrics: ovnkube_master_network_programming_ovn_duration_seconds_* and ovnkube_master_network_programming_duration_seconds_*.

Actual results:
Metrics not found.

Expected results:
Metrics found.


Additional info:
[1] Get the leader pod name from here: oc -n openshift-ovn-kubernetes get po -l app=ovnkube-master --field-selector spec.nodeName=$(oc -n openshift-ovn-kubernetes get cm ovn-kubernetes-master -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io\/leader}' | jq -r '.holderIdentity') -o jsonpath='{.items[0].metadata.name}'

Comment 3 Weibin Liang 2022-06-08 16:00:37 UTC
Tested and verified in 4.11.0-0.nightly-2022-06-06-201913

sh-4.4# curl 127.0.0.1:29102/metrics | grep ovnkube_master_network_programming_duration_seconds    
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP ovnkube_master_network_programming_duration_seconds The duration to apply network configuration for a kind (e.g. pod, service, networkpolicy). Configuration includes add, update and delete events for each kind.
# TYPE ovnkube_master_network_programming_duration_seconds histogram
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="0"} 0
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="0.25"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="0.5"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="1"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="2"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="3"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="4"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="5"} 58
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="6"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="7"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="8"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="9"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="10"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="11"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="12"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="13"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="14"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="15"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="16"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="17"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="18"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="19"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="20"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="21"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="22"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="23"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="24"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="25"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="26"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="27"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="28"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="29"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="30"} 59
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="31"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="32"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="33"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="34"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="35"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="36"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="37"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="38"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="39"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="40"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="41"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="42"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="43"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="44"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="45"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="46"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="47"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="48"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="49"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="50"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="51"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="52"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="53"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="54"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="55"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="56"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="57"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="58"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="59"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="60"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="65"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="70"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="75"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="80"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="85"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="90"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="95"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="100"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="105"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="110"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="115"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="120"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="150"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="180"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="210"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="240"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="270"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="300"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="330"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="360"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="390"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="420"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="+Inf"} 60
ovnkube_master_network_programming_duration_seconds_sum{kind="pod"} 35.72184365000002
ovnkube_master_network_programming_duration_seconds_count{kind="pod"} 60
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="0"} 0
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="0.25"} 14
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="0.5"} 14
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="1"} 14
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="2"} 14
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="3"} 14
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="4"} 14
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="5"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="6"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="7"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="8"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="9"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="10"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="11"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="12"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="13"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="14"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="15"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="16"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="17"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="18"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="19"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="20"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="21"} 16
100 62414    0 62414    0     0  59.5M      0 --:--:-- --:--:-- --:--:-- 59.5M
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="22"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="23"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="24"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="25"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="26"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="27"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="28"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="29"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="30"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="31"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="32"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="33"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="34"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="35"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="36"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="37"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="38"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="39"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="40"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="41"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="42"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="43"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="44"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="45"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="46"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="47"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="48"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="49"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="50"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="51"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="52"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="53"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="54"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="55"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="56"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="57"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="58"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="59"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="60"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="65"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="70"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="75"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="80"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="85"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="90"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="95"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="100"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="105"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="110"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="115"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="120"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="150"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="180"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="210"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="240"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="270"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="300"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="330"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="360"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="390"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="420"} 16
ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="+Inf"} 16
ovnkube_master_network_programming_duration_seconds_sum{kind="service"} 8.353540311000003
ovnkube_master_network_programming_duration_seconds_count{kind="service"} 16
sh-4.4#

Comment 4 Weibin Liang 2022-06-08 16:11:02 UTC
sh-4.4# curl 127.0.0.1:29102/metrics | grep ovnkube_master_network_programming_ovn_duration_seconds
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP ovnkube_master_network_programming_ovn_duration_seconds The duration for OVN to apply network configuration
# TYPE ovnkube_master_network_programming_ovn_duration_seconds histogram
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="0"} 0
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="0.25"} 10
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="0.5"} 10
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="1"} 10
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="2"} 10
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="3"} 10
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="4"} 10
ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="5"} 12

Comment 6 errata-xmlrpc 2022-08-10 11:14:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069


Note You need to log in before you can comment on or make changes to this bug.