Bug 1980357

This does look like a purely metrics related issue. Has the CU tried creating additional nodes once the alert starts firing? Does that fail? Looking at the code, it looks like the subnet allocation may still happen just fine.

Posted https://github.com/ovn-org/ovn-kubernetes/pull/2333 that should fix the issue. Also, I can confirm that the actual subnet allocation goes thru' just fine when you add additional nodes after the spurious alert.

Downstream fix: https://github.com/openshift/ovn-kubernetes/pull/605

ACK, Thank you!.

Verified this bug on 4.9.0-0.nightly-2021-07-29-103526

Reproduced this issue on 4.8.3

setup cluster with the following config with 4.8.3 version

~~~
networking:
  clusterNetwork:
  - cidr: 10.195.64.0/19 
    hostPrefix: 23 
  networkType: OVNKubernetes 
~~~

this can create at most 15 nodes. 

1. Create 3 master and 6 workers
2. after reboot all master worker
3. Check the alert with following command:

#token=`oc -n openshift-monitoring sa get-token prometheus-k8s`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-1  -- curl -k -H "Authorization: Bearer $token" https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?filter={alertname="V4SubnetAllocationThresholdExceeded"}
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   956  100   956    0     0  30838      0 --:--:-- --:--:-- --:--:-- 30838
[{"annotations":{"message":"More than 80% of v4 subnets available to assign to the nodes are allocated. Current v4 subnet allocation percentage is 112.5\n"},"endsAt":"2021-07-30T02:03:45.223Z","fingerprint":"7099016df0883a8c","receivers":[{"name":"Default"}],"startsAt":"2021-07-30T01:42:45.223Z","status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},"updatedAt":"2021-07-30T01:59:45.232Z","generatorURL":"https://prometheus-k8s-openshift-monitoring.apps.zzhaobug28.qe.devcluster.openshift.com/graph?g0.expr=ovnkube_master_allocated_v4_host_subnets+%2F+ovnkube_master_num_v4_host_subnets+%2A+100+%3E+80\u0026g0.tab=1","labels":{"alertname":"V4SubnetAllocationThresholdExceeded","container":"kube-rbac-proxy","endpoint":"metrics","instance":"10.0.179.203:9102","job":"ovnkube-master","namespace":"openshift-ovn-kubernetes","pod":"ovnkube-master-4x9z4","prometheus":"openshift-monitoring/k8s","service":"ovn-kubernetes-master","severity":"warning"}}]

4. Upgrade cluster to 4.9.0-0.nightly-2021-07-29-103526

5. Check step3 again, the alert disappeared. 

# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-1  -- curl -k -H "Authorization: Bearer $token" https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?filter={alertname="V4SubnetAllocationThresholdExceeded"}
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     3  100     3    0     0    100      0 --:--:-- --:--:-- --:--:--   100
[]

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759