Bug 1905647

In preparation for enabling consumption tracking, we should export a metric from all clusters that represents the sum of physical cores covered under the subscription / usage agreement.  We report min and max over 5m to establish a floor and ceiling for use (since scale down may take several minutes and we are willing to undercount to benefit users).

Physical cores are our best estimate of real cores in use on a node that is not labelled as infra (excluded via agreement from use by general purpose workload) or control plane (when control plane is considered unschedulable for general purpose workloads). A physical core is an unshared CPU as seen by the operating system, ignoring hyperthreading or virtualization.  In cloud environments a vCPU is often seen as a physical core and may vary based on instance size.

The counter metric will be used to gain experience in real world scenarios about the precision and failure modes of our current prometheus -> thanos aggregation (with or without direct write) in order to put error bounds on the precision of the behavior. In the long run counters located close to the cluster are the most resilient metric option for assessing usage since they can be interpolated even in the face of significant disruption to find average use.

The metric names will be

workload:capacity_physical_cpu_cores - current physical cores available for workloads, excluding infra and unschedulable masters

cluster:usage:workload:capacity_physical_cpu_cores:min:5m - minimum over 5m window

cluster:usage:workload:capacity_physical_cpu_cores:max:5m - maximum over 5m window

cluster:usage:workload:capacity_physical_cpu_core_seconds - recording rule converting the gauge into a counter

The last 3 will be exposed via telemetry to simplify usage billing.

Part of 4.7 to gain experience, may be backported to 4.6 as necessary.