Bug 1980357 - Getting the alert "V4SubnetAllocationThresholdExceeded" in newly installed cluster, Where subnet allocation is not more then 80%
Summary: Getting the alert "V4SubnetAllocationThresholdExceeded" in newly installed cl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.7
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: ---
: 4.9.0
Assignee: Aniket Bhat
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks: 1999638
TreeView+ depends on / blocked
 
Reported: 2021-07-08 13:12 UTC by Ashish Sharma
Modified: 2021-10-18 17:38 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1999638 (view as bug list)
Environment:
Last Closed: 2021-10-18 17:38:24 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift okd issues 594 0 None closed More than 80% of v4 subnets available to assign to the nodes are allocated. Current v4 subnet allocation percentage is 8... 2021-07-09 07:55:06 UTC
Github openshift ovn-kubernetes pull 609 0 None closed Merge 2021-07-18 2021-07-23 13:38:00 UTC
Github ovn-org ovn-kubernetes pull 2333 0 None open Fix duplicate incrementing of subnet allocation metric 2021-07-13 02:06:29 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:38:42 UTC

Description Ashish Sharma 2021-07-08 13:12:26 UTC
Description of problem:
Cluster Version:-OCP 4.7.18
Network Plugin:- OVN-kubernetes
Issue:-
Cu is using 10.195.64.0/19 cidr with hostprefix /24 which allows to add 32 nodes, He added 13 nodes which is not 80% of 32, still he is getting V4SubnetAllocationThresholdExceeded alert


 So his query is why he is getting this alert ,
metric for one master : ovnkube_master_allocated_v4_host_subnets shows allocated subnet is 26 allocated.





Version-Release number of selected component (if applicable):


How reproducible:

I have tried same metric query in quick lab in which i have added 6 nodes still i am getting 9 allocated subnet.



Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 4 Aniket Bhat 2021-07-12 16:46:16 UTC
This does look like a purely metrics related issue. Has the CU tried creating additional nodes once the alert starts firing? Does that fail? Looking at the code, it looks like the subnet allocation may still happen just fine.

Comment 5 Aniket Bhat 2021-07-13 02:07:51 UTC
Posted https://github.com/ovn-org/ovn-kubernetes/pull/2333 that should fix the issue. Also, I can confirm that the actual subnet allocation goes thru' just fine when you add additional nodes after the spurious alert.

Comment 11 Surya Seetharaman 2021-07-15 07:28:27 UTC
Downstream fix: https://github.com/openshift/ovn-kubernetes/pull/605

Comment 17 Ezequiel Hector Brizuela 2021-07-21 03:28:11 UTC
ACK, Thank you!.

Comment 20 zhaozhanqi 2021-07-30 04:41:00 UTC
Verified this bug on 4.9.0-0.nightly-2021-07-29-103526

Reproduced this issue on 4.8.3

setup cluster with the following config with 4.8.3 version

~~~
networking:
  clusterNetwork:
  - cidr: 10.195.64.0/19 
    hostPrefix: 23 
  networkType: OVNKubernetes 
~~~

this can create at most 15 nodes. 

1. Create 3 master and 6 workers
2. after reboot all master worker
3. Check the alert with following command:

#token=`oc -n openshift-monitoring sa get-token prometheus-k8s`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-1  -- curl -k -H "Authorization: Bearer $token" https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?filter={alertname="V4SubnetAllocationThresholdExceeded"}
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   956  100   956    0     0  30838      0 --:--:-- --:--:-- --:--:-- 30838
[{"annotations":{"message":"More than 80% of v4 subnets available to assign to the nodes are allocated. Current v4 subnet allocation percentage is 112.5\n"},"endsAt":"2021-07-30T02:03:45.223Z","fingerprint":"7099016df0883a8c","receivers":[{"name":"Default"}],"startsAt":"2021-07-30T01:42:45.223Z","status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},"updatedAt":"2021-07-30T01:59:45.232Z","generatorURL":"https://prometheus-k8s-openshift-monitoring.apps.zzhaobug28.qe.devcluster.openshift.com/graph?g0.expr=ovnkube_master_allocated_v4_host_subnets+%2F+ovnkube_master_num_v4_host_subnets+%2A+100+%3E+80\u0026g0.tab=1","labels":{"alertname":"V4SubnetAllocationThresholdExceeded","container":"kube-rbac-proxy","endpoint":"metrics","instance":"10.0.179.203:9102","job":"ovnkube-master","namespace":"openshift-ovn-kubernetes","pod":"ovnkube-master-4x9z4","prometheus":"openshift-monitoring/k8s","service":"ovn-kubernetes-master","severity":"warning"}}]

4. Upgrade cluster to 4.9.0-0.nightly-2021-07-29-103526

5. Check step3 again, the alert disappeared. 

# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-1  -- curl -k -H "Authorization: Bearer $token" https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?filter={alertname="V4SubnetAllocationThresholdExceeded"}
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     3  100     3    0     0    100      0 --:--:-- --:--:-- --:--:--   100
[]

Comment 28 errata-xmlrpc 2021-10-18 17:38:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.