Description of problem: Configuration duration monitoring is not enabled by default. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Curl the metrics endpoint for the ovnkube-master leader on port 9102. [1] 2. Search for 2 Prometheus bucket metrics: ovnkube_master_network_programming_ovn_duration_seconds_* and ovnkube_master_network_programming_duration_seconds_*. Actual results: Metrics not found. Expected results: Metrics found. Additional info: [1] Get the leader pod name from here: oc -n openshift-ovn-kubernetes get po -l app=ovnkube-master --field-selector spec.nodeName=$(oc -n openshift-ovn-kubernetes get cm ovn-kubernetes-master -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io\/leader}' | jq -r '.holderIdentity') -o jsonpath='{.items[0].metadata.name}'
Tested and verified in 4.11.0-0.nightly-2022-06-06-201913 sh-4.4# curl 127.0.0.1:29102/metrics | grep ovnkube_master_network_programming_duration_seconds % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0# HELP ovnkube_master_network_programming_duration_seconds The duration to apply network configuration for a kind (e.g. pod, service, networkpolicy). Configuration includes add, update and delete events for each kind. # TYPE ovnkube_master_network_programming_duration_seconds histogram ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="0"} 0 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="0.25"} 58 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="0.5"} 58 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="1"} 58 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="2"} 58 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="3"} 58 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="4"} 58 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="5"} 58 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="6"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="7"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="8"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="9"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="10"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="11"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="12"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="13"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="14"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="15"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="16"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="17"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="18"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="19"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="20"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="21"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="22"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="23"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="24"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="25"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="26"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="27"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="28"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="29"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="30"} 59 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="31"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="32"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="33"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="34"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="35"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="36"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="37"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="38"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="39"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="40"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="41"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="42"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="43"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="44"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="45"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="46"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="47"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="48"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="49"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="50"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="51"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="52"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="53"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="54"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="55"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="56"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="57"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="58"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="59"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="60"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="65"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="70"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="75"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="80"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="85"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="90"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="95"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="100"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="105"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="110"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="115"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="120"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="150"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="180"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="210"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="240"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="270"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="300"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="330"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="360"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="390"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="420"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="pod",le="+Inf"} 60 ovnkube_master_network_programming_duration_seconds_sum{kind="pod"} 35.72184365000002 ovnkube_master_network_programming_duration_seconds_count{kind="pod"} 60 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="0"} 0 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="0.25"} 14 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="0.5"} 14 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="1"} 14 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="2"} 14 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="3"} 14 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="4"} 14 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="5"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="6"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="7"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="8"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="9"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="10"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="11"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="12"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="13"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="14"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="15"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="16"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="17"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="18"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="19"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="20"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="21"} 16 100 62414 0 62414 0 0 59.5M 0 --:--:-- --:--:-- --:--:-- 59.5M ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="22"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="23"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="24"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="25"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="26"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="27"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="28"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="29"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="30"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="31"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="32"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="33"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="34"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="35"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="36"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="37"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="38"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="39"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="40"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="41"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="42"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="43"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="44"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="45"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="46"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="47"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="48"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="49"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="50"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="51"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="52"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="53"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="54"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="55"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="56"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="57"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="58"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="59"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="60"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="65"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="70"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="75"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="80"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="85"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="90"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="95"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="100"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="105"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="110"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="115"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="120"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="150"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="180"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="210"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="240"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="270"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="300"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="330"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="360"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="390"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="420"} 16 ovnkube_master_network_programming_duration_seconds_bucket{kind="service",le="+Inf"} 16 ovnkube_master_network_programming_duration_seconds_sum{kind="service"} 8.353540311000003 ovnkube_master_network_programming_duration_seconds_count{kind="service"} 16 sh-4.4#
sh-4.4# curl 127.0.0.1:29102/metrics | grep ovnkube_master_network_programming_ovn_duration_seconds % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0# HELP ovnkube_master_network_programming_ovn_duration_seconds The duration for OVN to apply network configuration # TYPE ovnkube_master_network_programming_ovn_duration_seconds histogram ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="0"} 0 ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="0.25"} 10 ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="0.5"} 10 ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="1"} 10 ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="2"} 10 ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="3"} 10 ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="4"} 10 ovnkube_master_network_programming_ovn_duration_seconds_bucket{le="5"} 12
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069