Bug 2017983

Summary: Upgrade from OCP 4.7.31/32/16 to 4.7.33 is failing with MULTUS IPVLAN
Product: OpenShift Container Platform Reporter: Vishvranjan Mishra <vismishr>
Component: NetworkingAssignee: Douglas Smith <dosmith>
Networking sub component: multus QA Contact: Weibin Liang <weliang>
Status: CLOSED DUPLICATE Docs Contact:
Severity: urgent    
Priority: urgent CC: ngirard
Version: 4.7   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-28 17:47:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vishvranjan Mishra 2021-10-27 20:18:45 UTC 
Description of problem:
Issue: Upgrade from OCP 4.7.31/32/16 to 4.7.33 is failing MULTUS IPVLAN
Upgrade from 4.7.16 to 4.7.31/32 was successful.



Version-Release number of selected component (if applicable):
4.7.33


How reproducible:
Upgrade from OCP 4.7.31/32/16 to 4.7.33 is failing with MULTUS IPVLAN



Steps to Reproduce:
Upgrade from OCP 4.7.31/32/16 to 4.7.33 is failing with MULTUS IPVLAN

Actual results:
Failed to create sandbox



Expected results:
It should upgrade with MULTUS
Comment 2 Douglas Smith 2021-10-28 12:50:38 UTC 
I believe that the customer may be experiencing this bug: 

To work around it, you may apply the RBAC rules manually, create a file called `wb-rbac.yml` with these contents: https://bugzilla.redhat.com/show_bug.cgi?id=1990932

```
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: multus-whereabouts
  namespace: openshift-multus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: whereabouts-cni
subjects:
- kind: ServiceAccount
  name: multus
  namespace: openshift-multus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: openshift-multus
  name: whereabouts-cni
rules:
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - '*'
```

And then `oc create -f wb-rbac.yml` -- this should give the Whereabouts IPAM CNI plugin access to the lease CRs.

You may also wish to restart the pods referencing net-attach-defs with Whereabouts IPAM, otherwise, it should clear on the next crash loop.
Comment 3 Douglas Smith 2021-10-28 12:51:23 UTC 
Of note, the fix for this was merged in https://github.com/openshift/cluster-network-operator/pull/1186/files for 4.7.z, however, I'm unsure of which z-stream this appears in.
Comment 4 Neil Girard 2021-10-28 12:58:35 UTC 
Looks like it was in errata for 4.7.34.  Customer is updating to 4.7.33 so the need to go up one more.
Comment 5 Douglas Smith 2021-10-28 14:28:29 UTC 
regarding my comment: 

> You may also wish to restart the pods referencing net-attach-defs with Whereabouts IPAM, otherwise, it should clear on the next crash loop.


I mean the pods that are the workload pods, NOT the pods in the `openshift-multus` namespace.

That is, the customer pods, the pods that have the "k8s.v1.cni.cncf.io/networks" annotation.

This is NOT required, but, would speed up the process instead of waiting for the crashloopbackoff timeout. So, completely optional.
Comment 6 Douglas Smith 2021-10-28 17:47:53 UTC 
It's my current diagnosis that this was a dupe of 1990932.

*** This bug has been marked as a duplicate of bug 1990932 ***