Description of problem: Cluster Version:-OCP 4.7.18 Network Plugin:- OVN-kubernetes Issue:- Cu is using 10.195.64.0/19 cidr with hostprefix /24 which allows to add 32 nodes, He added 13 nodes which is not 80% of 32, still he is getting V4SubnetAllocationThresholdExceeded alert So his query is why he is getting this alert , metric for one master : ovnkube_master_allocated_v4_host_subnets shows allocated subnet is 26 allocated. Version-Release number of selected component (if applicable): How reproducible: I have tried same metric query in quick lab in which i have added 6 nodes still i am getting 9 allocated subnet. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
This does look like a purely metrics related issue. Has the CU tried creating additional nodes once the alert starts firing? Does that fail? Looking at the code, it looks like the subnet allocation may still happen just fine.
Posted https://github.com/ovn-org/ovn-kubernetes/pull/2333 that should fix the issue. Also, I can confirm that the actual subnet allocation goes thru' just fine when you add additional nodes after the spurious alert.
Downstream fix: https://github.com/openshift/ovn-kubernetes/pull/605
ACK, Thank you!.
Verified this bug on 4.9.0-0.nightly-2021-07-29-103526 Reproduced this issue on 4.8.3 setup cluster with the following config with 4.8.3 version ~~~ networking: clusterNetwork: - cidr: 10.195.64.0/19 hostPrefix: 23 networkType: OVNKubernetes ~~~ this can create at most 15 nodes. 1. Create 3 master and 6 workers 2. after reboot all master worker 3. Check the alert with following command: #token=`oc -n openshift-monitoring sa get-token prometheus-k8s` # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-1 -- curl -k -H "Authorization: Bearer $token" https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?filter={alertname="V4SubnetAllocationThresholdExceeded"} % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 956 100 956 0 0 30838 0 --:--:-- --:--:-- --:--:-- 30838 [{"annotations":{"message":"More than 80% of v4 subnets available to assign to the nodes are allocated. Current v4 subnet allocation percentage is 112.5\n"},"endsAt":"2021-07-30T02:03:45.223Z","fingerprint":"7099016df0883a8c","receivers":[{"name":"Default"}],"startsAt":"2021-07-30T01:42:45.223Z","status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},"updatedAt":"2021-07-30T01:59:45.232Z","generatorURL":"https://prometheus-k8s-openshift-monitoring.apps.zzhaobug28.qe.devcluster.openshift.com/graph?g0.expr=ovnkube_master_allocated_v4_host_subnets+%2F+ovnkube_master_num_v4_host_subnets+%2A+100+%3E+80\u0026g0.tab=1","labels":{"alertname":"V4SubnetAllocationThresholdExceeded","container":"kube-rbac-proxy","endpoint":"metrics","instance":"10.0.179.203:9102","job":"ovnkube-master","namespace":"openshift-ovn-kubernetes","pod":"ovnkube-master-4x9z4","prometheus":"openshift-monitoring/k8s","service":"ovn-kubernetes-master","severity":"warning"}}] 4. Upgrade cluster to 4.9.0-0.nightly-2021-07-29-103526 5. Check step3 again, the alert disappeared. # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-1 -- curl -k -H "Authorization: Bearer $token" https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?filter={alertname="V4SubnetAllocationThresholdExceeded"} % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3 100 3 0 0 100 0 --:--:-- --:--:-- --:--:-- 100 []
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759