Bug 2080894

Description of problem:
We are using labelled (vector) metrics to export
info and if the label values change, we still
export the old labels values which confuses users
because they have no way of determining which labelled
metric is the latest.

We must reset each vector metric prior to using it to
ensure we are only exporting the current state and not
old states which should have been scraped previously due
to our usage of labels that can be dynamically updated
during runtime.

This is occurring in the OVN metrics exported by
ovnkube-node.

Follow on work is needed to fix OVS metrics that are
used by ovs-exporter.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Get OVN Northbound database leader (use cluster/status command from ovn-nbctl)
2. Login to the ovnkube-node container on that node.
3. Curl the metrics endpoint and grep for ovn_db_cluster_server_role. Note the server_role label is leader.
4. On the same pod, repeatedly kill the nbdb container process only until the ovn_db_cluster_server_role metric changes it label 'server_role' from leader to 'cluster follower'. Do not kill the whole pod. Exec into the container continuously and send a sigterm to the container nbdb process.
5. The metric ovn_db_cluster_server_role has two values - it still exposes 'server_role' as leader but also as 'cluster follower'.

Note: it maybe very awkward to get the northbound database to switch from leader to 'cluster follower'. If you can freeze the process instead of killing it, you may have a better chance of it changing.

Actual results:
There are two values for ovn_db_cluster_server_role

Expected results:
There should only be one value or entry for ovn_db_cluster_server_role

Additional info:
Contact Nadia Pinaeva for further information if you cannot reproduce. I maybe on PTO.