Bug 1746329

Summary: [sriov] NetworkAttachmentDefinition always be recreated even if it is exist after sriov operater is recreated
Product: OpenShift Container Platform Reporter: zhaozhanqi <zzhao>
Component: NetworkingAssignee: Peng Liu <pliu>
Status: CLOSED ERRATA QA Contact: zhaozhanqi <zzhao>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.2.0CC: aos-bugs, zshi
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 06:38:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description zhaozhanqi 2019-08-28 08:01:55 UTC
Description of problem:
NetworkAttachmentDefinition always be recreated after sriov operater pod is recreated
see logs:
{"level":"error","ts":1566973127.122615,"logger":"controller_sriovnetwork","msg":"Couldn't create NetworkAttachmentDefinition","Request.Namespace":"sriov-network-operator","Request.Name":"example-sriovnetwork","Namespace":"z1","Name":"example-sriovnetwork","error":"network-attachment-definitions.k8s.cni.cncf.io \"example-sriovnetwork\" already exists","stacktrace":"github.com/openshift/sriov-network-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/openshift/sriov-network-operator/pkg/controller/sriovnetwork.(*ReconcileSriovNetwork).Reconcile\n\t/go/src/github.com/openshift/sriov-network-operator/pkg/controller/sriovnetwork/sriovnetwork_controller.go:185\ngithub.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1566973127.1228464,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"sriovnetwork-controller","request":"sriov-network-operator/example-sriovnetwork","error":"network-attachment-definitions.k8s.cni.cncf.io \"example-sriovnetwork\" already exists","stacktrace":"github.com/openshift/sriov-network-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1566973290.963193,"logger":"controller_sriovnetwork","msg":"Reconciling SriovNetwork","Request.Namespace":"sriov-network-operator","Request.Name":"example-sriovnetwork"}
{"level":"info","ts":1566973290.9680672,"logger":"controller_sriovnetwork.renderNetAttDef","msg":"Start to render SRIOV CNI NetworkAttachementDefinition"}
manifest {"apiVersion":"k8s.cni.cncf.io/v1","kind":"NetworkAttachmentDefinition","metadata":{"annotations":{"k8s.v1.cni.cncf.io/resourceName":"openshift.io/intelnics1"},"name":"example-sriovnetwork","namespace":"z1"},"spec":{"config":"{\"cniVersion\":\"0.3.1\",\"name\":\"sriov-net\",\"type\":\"sriov\",\"spoofchk\":\"on\",\"trust\":\"off\",\"vlan\":0,\"ipam\":{\"type\":\"host-local\",\"subnet\":\"10.56.217.0/24\",\"rangeStart\":\"10.56.217.171\",\"rangeEnd\":\"10.56.217.181\",\"routes\":[{\"dst\":\"0.0.0.0/0\"}],\"gateway\":\"10.56.217.1\"}}\n"}}
{"level":"info","ts":1566973290.9699564,"logger":"controller_sriovnetwork","msg":"Creating a new NetworkAttachmentDefinition","Request.Namespace":"sriov-network-operator","Request.Name":"example-sriovnetwork","Namespace":"z1","Name":"example-sriovnetwork"}
{"level":"error","ts":1566973291.1118793,"logger":"controller_sriovnetwork","msg":"Couldn't create NetworkAttachmentDefinition","Request.Namespace":"sriov-network-operator","Request.Name":"example-sriovnetwork","Namespace":"z1","Name":"example-sriovnetwork","error":"network-attachment-definitions.k8s.cni.cncf.io \"example-sriovnetwork\" already exists","stacktrace":"github.com/openshift/sriov-network-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/openshift/sriov-network-operator/pkg/controller/sriovnetwork.(*ReconcileSriovNetwork).Reconcile\n\t/go/src/github.com/openshift/sriov-network-operator/pkg/controller/sriovnetwork/sriovnetwork_controller.go:185\ngithub.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1566973291.112095,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"sriovnetwork-controller","request":"sriov-network-operator/example-sriovnetwork","error":"network-attachment-definitions.k8s.cni.cncf.io \"example-sriovnetwork\" already exists","stacktrace":"github.com/openshift/sriov-network-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1566973455.3132722,"logger":"controller_caconfig","msg":"Reconciling CA config map","Request.Namespace":"sriov-network-operator","Request.Name":"openshift-service-ca"}
Collaps


Version-Release number of selected component (if applicable):
quay.io/openshift-release-dev/ocp-v4.0-art-dev:v4.2.0-201908270619-ose-sriov-network-operator

How reproducible:
always

Steps to Reproduce:
1. install sriov operator
2. Create sriovnetwork CR to create one NetworkAttachmentDefinition in namespaces z1

 oc get sriovnetworks.sriovnetwork.openshift.io example-sriovnetwork -o yaml
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  creationTimestamp: "2019-08-27T09:30:26Z"
  generation: 1
  name: example-sriovnetwork
  namespace: sriov-network-operator
  resourceVersion: "4480917"
  selfLink: /apis/sriovnetwork.openshift.io/v1/namespaces/sriov-network-operator/sriovnetworks/example-sriovnetwork
  uid: 4d610e2c-c8ad-11e9-9801-3cfdfeba0aa4
spec:
  ipam: |
    {
      "type": "host-local",
      "subnet": "10.56.217.0/24",
      "rangeStart": "10.56.217.171",
      "rangeEnd": "10.56.217.181",
      "routes": [{
        "dst": "0.0.0.0/0"
      }],
      "gateway": "10.56.217.1"
    }
  networkNamespace: z1
  resourceName: intelnics1
  spoofChk: true
  trust: false
  vlan: 0
  

3. Delete the sriov operator pod to make it recreated
4. Check the logs of new created sriov operator pod

Actual results:

see $description

Expected results:

no need to recreated the exist NAD

Additional info:

Comment 2 zhaozhanqi 2019-09-04 09:23:02 UTC
Verified this bug on quay.io/openshift-release-dev/ocp-v4.0-art-dev:v4.2.0-201909020729-ose-sriov-network-operator

Comment 3 errata-xmlrpc 2019-10-16 06:38:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922