Bug 1780611

Summary: Multus does not provide a clear error message or cause when the default network is broken
Product: OpenShift Container Platform Reporter: Ben Bennett <bbennett>
Component: NetworkingAssignee: Tomofumi Hayashi <tohayash>
Networking sub component: multus QA Contact: Weibin Liang <weliang>
Status: CLOSED WONTFIX Docs Contact:
Severity: urgent    
Priority: unspecified CC: bbennett, ccoleman, cdc, dosmith, nagrawal, tohayash, wking, zzhao
Version: 4.3.0   
Target Milestone: ---   
Target Release: 4.3.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1777040 Environment:
Last Closed: 2020-07-01 13:05:00 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1777038, 1777040    
Bug Blocks:    

Description Ben Bennett 2019-12-06 14:07:07 UTC 
+++ This bug was initially created as a clone of Bug #1777040 +++

In the linked bug we pushed a cluster past what openshift-sdn could support, but multus only displayed a generic error in the Kubelet status ("CNI default network not found").  This is a pretty critical message to get right for an admin to know where to look, so there are a few improvements that need to be made:

1. The message should more clearly indicate what the heck a default CNI network means - I suggest something like "Default networking provider for this node has not been registered, unable to assign IPs to pods"

2. It would be good to consider whether multus has enough info to tell us where we should be looking next (the pod or provider) - if not, that's ok, but this is a usability chokepoint.  Whether that's changes to CNI (so that partial failure could be reported) or whether it's something the network plugin should be telling multus about at a kube level, or whether it's some other mechanism, it doesn't matter so long as you don't have to infer *where* the default network provider is.

3. There should be a metric and alert for this sort of scenario. In the linked bug the networking provider should be reporting a capacity error (can't allocate any more subnets).  But multus should probably be reporting a metric on whether the default network is available, and if more detail is available, why.

Setting to urgent because the message is not explicable to a normal admin, and a normal admin could hit it.  It should be crystal clear that the admin needs to go debug the cluster networking provider and where that is.

--- Additional comment from Tomofumi Hayashi on 2019-12-01 23:29:22 EST ---

Is the error message actually "Missing CNI default network", instead of "CNI default network not found"?

--- Additional comment from Tomofumi Hayashi on 2019-12-04 09:09:46 EST ---

https://github.com/cri-o/ocicni/pull/66