Description of problem (please be detailed as possible and provide log snippests): The ODF must-gather does not collect NetworkAttachmentDefinitions that affect Multus. The must-gather should be updated to collect NetworkAttachmentDefinition resources from the "default" and "openshift-storage" namespaces. Version of all relevant components (if applicable): I would hope this can be part of an upcoming 4.13.z release. Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? No. Is there any workaround available to the best of your knowledge? OCP must-gather collects NetworkAttachmentDefinitions Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? Yes. Can this issue reproduce from the UI? N/A If this is a regression, please provide more details to justify this: N/A Steps to Reproduce: It is not necessary to install ODF with multus enabled to repro! First, create these two NetworkAttachmentDefinitions: apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: cluster-net namespace: openshift-storage spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.30.0/24" } }' apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: cluster-net spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.30.0/24" } }' Next, collect an ODF must-gather Both NADs should be collected but are not. The first NAD is in the openshift-storage NS, and the second is in the default NS. NADs from other NSes do not need to be collected for ODF.
I see this is merged into the m-g main codebase and is in 4.14. @etamir would you like to target a 4.13.z backport as well? What about 4.12.z for any support exceptions?
@ypadia would you be able to take the action to make sure backport BZs get created for 4.13.z and 4.12.z? I'd like to get this off of my todo list to keep focusing on Rook BZs. Thanks :)
Yati, please create clones for 4.13/4.12. Reach out to Sunil for adding the backports to the respective z-streams. Oded, please provide qa_ack
Yati, please create clones bugs for 4.13/4.12
Bug fixed. SetUp: ODF Version: odf-operator.v4.14.0-115.stable OCP Version: 4.14.0-0.nightly-2023-08-11-055332 Platform: Vsphere Test Process: 1. Collect mg: $ oc adm must-gather --image=quay.io/rhceph-dev/ocs-must-gather:latest-4.14 2. Check log content: $ pwd /tmp/tmphpy1wpbo_ocs_logs/oviner21-14/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-427ab3d9756b1096dc2afbf4126ac0e1ce2d367b8b302f584d1403129ae84714/namespaces/all $ ls -l | grep net -rw-r--r--. 1 oviner oviner 3042 Aug 24 15:39 desc_net_attach_def_all_ns -rw-r--r--. 1 oviner oviner 1852 Aug 24 15:39 get_yaml_net_attach_def_all_ns $ cat desc_net_attach_def_all_ns | more Name: cluster-net Namespace: default Labels: <none> Annotations: <none> API Version: k8s.cni.cncf.io/v1 Kind: NetworkAttachmentDefinition Metadata: Creation Timestamp: 2023-08-24T08:36:26Z Generation: 1 Managed Fields: API Version: k8s.cni.cncf.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: .: f:config: Manager: kubectl-create Operation: Update Time: 2023-08-24T08:36:26Z Resource Version: 366999 UID: 3799c619-2774-4c2f-a634-a89a37d556f8 Spec: Config: { "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.30.0/24" } } Events: <none> $ cat get_yaml_net_attach_def_all_ns | more apiVersion: v1 items: - apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: creationTimestamp: "2023-08-24T08:36:26Z" generation: 1 name: cluster-net namespace: default resourceVersion: "366999" uid: 3799c619-2774-4c2f-a634-a89a37d556f8 spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.30.0/24" } }' - apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: creationTimestamp: "2023-08-24T08:36:26Z" generation: 1 name: public-net namespace: default resourceVersion: "366998" uid: 22a5a8e2-c580-4e10-9eb2-b12b92bffffd spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.20.0/24" } }' - apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: creationTimestamp: "2023-08-24T12:24:15Z" generation: 1
The NAD collection code appears twice [on release-4.14 branch]. https://github.com/red-hat-storage/odf-must-gather/blob/release-4.14/must-gather/collection-scripts/gather_namespaced_resources#L229-L239 I opened a new PR to fix it https://github.com/red-hat-storage/odf-must-gather/pull/60 SetUp: ODF Version: odf-operator.v4.14.0-126.stable OCP Version: 4.14.0-0.nightly-2023-09-02-132842 Platform: BM Test Process: 1. Collect mg: $ oc adm must-gather --image=quay.io/rhceph-dev/ocs-must-gather:latest-4.14 2.Check NAD oviner:all$ pwd namespaces/all oviner:all$ ls | grep net desc_net_attach_def_all_ns get_yaml_net_attach_def_all_ns
Oded, I closed the PR. You need to open a PR on main branch and not release-4.14 dirtectly. Can you please open a new PR against main branch?
The code that performs the NAD collection is duplicated. I also checked the image itself. $ podman pull quay.io/rhceph-dev/ocs-must-gather:latest-4.14 $ podman run --entrypoint /bin/bash -it quay.io/rhceph-dev/ocs-must-gather:latest-4.14 [root@eafcd3003a6d /]# [root@eafcd3003a6d bin]# pwd /usr/bin [root@eafcd3003a6d bin]# cat gather_namespaced_resources ... ... ... echo "collecting network-attachment-definitions of oc get network-attachment-definitions all namespaces" | tee -a "${BASE_COLLECTION_PATH}/gather-debug.log" if [ -n "$(oc get network-attachment-definitions --no-headers -A | awk '{print $1}')" ]; then { oc get network-attachment-definitions -oyaml --all-namespaces; } >>"${BASE_COLLECTION_PATH}/namespaces/all/get_yaml_net_attach_def_all_ns" { oc describe network-attachment-definitions --all-namespaces; } >>"${BASE_COLLECTION_PATH}/namespaces/all/desc_net_attach_def_all_ns" fi echo "collecting network-attachment-definitions of oc get network-attachment-definitions all namespaces" | tee -a "${BASE_COLLECTION_PATH}/gather-debug.log" if [ "$(oc get network-attachment-definitions --no-headers -A | awk '{print $1}')" != "" ] ; then { oc get network-attachment-definitions -oyaml --all-namespaces; } >> "${BASE_COLLECTION_PATH}/namespaces/all/get_yaml_net_attach_def_all_ns" { oc describe network-attachment-definitions --all-namespaces; } >> "${BASE_COLLECTION_PATH}/namespaces/all/desc_net_attach_def_all_ns" fi
Build was done before the PR was merged, next build must contain the changes.
bug fixed: SetUp: OCP Version: 4.14.0-0.nightly-2023-09-26-124507 ODF Version: odf-operator.v4.14.0-139.stable platform: Vsphere Test process: 1.Install OCP4.14+ODF4.14 [without multus] 2.Create NAD: --- apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: public-net namespace: default labels: {} annotations: {} spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.20.0/24" } }' --- apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: cluster-net namespace: default labels: {} annotations: {} spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.30.0/24" } }' 3.Collect MG: $ oc adm must-gather --image=quay.io/rhceph-dev/ocs-must-gather:latest-4.14 4.Check content: oviner:all$ pwd .../namespaces/all oviner:all$ ls | grep net desc_net_attach_def_all_ns get_yaml_net_attach_def_all_ns $ cat desc_net_attach_def_all_ns Name: cluster-net Namespace: default Labels: <none> Annotations: <none> API Version: k8s.cni.cncf.io/v1 Kind: NetworkAttachmentDefinition Metadata: Creation Timestamp: 2023-09-27T15:14:15Z Generation: 1 Managed Fields: API Version: k8s.cni.cncf.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: .: f:config: Manager: kubectl-create Operation: Update Time: 2023-09-27T15:14:15Z Resource Version: 158138 UID: ea10451a-53d2-4c79-9cca-e87f93db05b8 Spec: Config: { "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.30.0/24" } } Events: <none> Name: public-net Namespace: default Labels: <none> Annotations: <none> API Version: k8s.cni.cncf.io/v1 Kind: NetworkAttachmentDefinition Metadata: Creation Timestamp: 2023-09-27T15:14:15Z Generation: 1 Managed Fields: API Version: k8s.cni.cncf.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: .: f:config: Manager: kubectl-create Operation: Update Time: 2023-09-27T15:14:15Z Resource Version: 158135 UID: af9ef2af-2842-445e-88c2-e70e07423148 Spec: Config: { "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.20.0/24" } } Events: <none> $ cat get_yaml_net_attach_def_all_ns apiVersion: v1 items: - apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: creationTimestamp: "2023-09-27T15:14:15Z" generation: 1 name: cluster-net namespace: default resourceVersion: "158138" uid: ea10451a-53d2-4c79-9cca-e87f93db05b8 spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.30.0/24" } }' - apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: creationTimestamp: "2023-09-27T15:14:15Z" generation: 1 name: public-net namespace: default resourceVersion: "158135" uid: af9ef2af-2842-445e-88c2-e70e07423148 spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.20.0/24" } }' kind: List metadata: resourceVersion: "" selfLink: ""
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.14.0 security, enhancement & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6832