Since this can bring down user apps i consider this to block 4.2 to 4.3 upgrades going to customer.
Created attachment 1652538 [details] IP address allocation information from worker node
@huirwang I saw you verified this bug in 4.3.0-0.nightly-2020-01-14-014012. but https://bugzilla.redhat.com/show_bug.cgi?id=1787635#c5 is using 4.3.0-0.nightly-2020-01-13-223759 Please help check the above version if it's the fixed version. and if so. please have a try again if it can be reproduced and paste the logs of all workers. thanks
To get better SDN debugging with current 4.3 you'd need to bump the log level up to 5. If https://github.com/openshift/sdn/pull/78 merges then you should get more useful pod create/delete logging at the default log level.
One question that was raised was if Multus requires there to be a net namespace for a pod in order to delegate the CNI DEL call to another CNI plugin (especially: openshift-sdn) My findings show that Multus only requires the netns to be present during a CNI ADD call, in particular it requires the netns before it attempts to add an network interface to namespace using `LinkByName()` from the netlink library. Multus does look for the netns during a CNI DEL, but, its absence is tolerated. Multus continues execution and continues passing the CNI DEL to the delegated plugin (in this case, openshift-sdn) Additionally, another question has been raised about an error which reads as: ``` Jan 16 10:26:08 ip-10-0-146-171 crio[596572]: 2020-01-16T10:26:08Z [error] Multus: error unsetting the networks status: SetNetworkStatus: [...snipped...] ``` In this case, Multus just notes the error in the log and continues (multus.go@HEAD:583) ``` // error happen but continue to delete logging.Errorf("Multus: error unsetting the networks status: %v", err) ``` This improvement was made upstream in https://github.com/intel/multus-cni/pull/311 -- and this code exists in Multus CNI versions in OpenShift 4.2 and 4.3
Can we retest on a new cluster with kubelet logging set to v4 and crio in info mode? You can hop onto a node using oc debug/node and modify /etc/crio/crio.conf. Uncomment the log_level line and set the value to info. You can then systemctl reload crio and it will start logging at info level without restarting. If we get these logs, we can see check if kubelet is calling stop pod sandbox and crio is getting it to call into the networking cleanup code.
There were two cases where we were leaking and this bug covered both. I split https://bugzilla.redhat.com/show_bug.cgi?id=1792533 off to cover the kubelet restart leak. The case where we would leak on a reboot is covered by this bug, and that seems to have been completely resolved.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0391