Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem (please be detailed as possible and provide log snippests): Noobaa-DB Failed mount on cluster with multus Version of all relevant components (if applicable): OCP Version: 4.16.0-0.nightly-2024-05-16-092402 ODF Version: odf-operator.v4.16.0-95.stable Provider: BM Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1.Deploy OCP4.16 [4.16.0-0.nightly-2024-05-16-092402] 2.Install ODF4.16 [odf-operator.v4.16.0-95.stable] 3.Craete storagecluster with multus [public-net + cluster-net] 4.Check noobaa-db pod [stuck on ContainerCreating state] Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 85s default-scheduler Successfully assigned openshift-storage/noobaa-db-pg-0 to argo006.ceph.redhat.com Warning FailedMount 49s kubelet MountVolume.MountDevice failed for volume "pvc-79875fc6-2e0a-47e5-9251-a288337fda04" : rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e7baa6a2-b5fa-497b-9239-a8b62bb67a23 already exists 5.Ceph status is ok: sh-5.1$ ceph -s cluster: id: 93bf131c-e165-49f2-ae09-9168d05f1d20 health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 5d) mgr: a(active, since 5d), standbys: b mds: 1/1 daemons up, 1 hot standby osd: 3 osds: 3 up (since 5d), 3 in (since 5d) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 12 pools, 169 pgs objects: 429 objects, 799 KiB usage: 1.3 GiB used, 2.7 TiB / 2.7 TiB avail pgs: 169 active+clean io: client: 1.3 KiB/s rd, 170 B/s wr, 2 op/s rd, 0 op/s wr 6.Storagecluster is Progressing state $ oc get storageclusters.ocs.openshift.io NAME AGE PHASE EXTERNAL CREATED AT VERSION ocs-storagecluster 5d Progressing 2024-05-17T13:24:11Z 4.16.0 Type: Available Last Heartbeat Time: 2024-05-22T14:05:08Z Last Transition Time: 2024-05-17T13:24:12Z Message: Waiting on Nooba instance to finish initialization Reason: NoobaaInitializing Status: True Actual results: Expected results: Additional info: mg: http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-2282543/
Seems like a problem with the PVC not with NooBaa... Should be checked with CSI/Ceph
Nimrod, The ceph stautus is OK and pvc in bound state $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE db-noobaa-db-pg-0 Bound pvc-79875fc6-2e0a-47e5-9251-a288337fda04 50Gi RWO ocs-storagecluster-ceph-rbd <unset> 5d The cluster is running now. https://url.corp.redhat.com/5c83e5c
MountVolume.MountDevice failed is not a NooBaa or a PSQL error. This is a CSI error and it can be either through our CSI or Rook.
Hi Baline, I added NodeNetworkConfigurationPolicy to odf4.16 deployment and the issue reproduced Can you check my procedure? https://docs.google.com/document/d/1n6QNPscKpe1Bz08T6D00dl7fEbzY15QEsOYrbP_uehg/edit Procedure: https://bugzilla.redhat.com/show_bug.cgi?id=2282543 1.Deploy OCP4.16 on BM 2.Install LSO4.14 3.Install NMstate4.14 operator and create instance 4.Install ODF4.16 $ oc get csv -A NAMESPACE NAME DISPLAY VERSION REPLACES PHASE openshift-local-storage local-storage-operator.v4.15.0-202405070739 Local Storage 4.15.0-202405070739 local-storage-operator.v4.14.0-202311031050 Succeeded openshift-nmstate kubernetes-nmstate-operator.4.15.0-202405070739 Kubernetes NMState Operator 4.15.0-202405070739 kubernetes-nmstate-operator.4.14.0-202311021650 Succeeded openshift-operator-lifecycle-manager packageserver Package Server 0.0.1-snapshot Succeeded openshift-storage mcg-operator.v4.16.0-108.stable NooBaa Operator 4.16.0-108.stable Succeeded openshift-storage ocs-client-operator.v4.16.0-108.stable OpenShift Data Foundation Client 4.16.0-108.stable Succeeded openshift-storage ocs-operator.v4.16.0-108.stable OpenShift Container Storage 4.16.0-108.stable Succeeded openshift-storage odf-csi-addons-operator.v4.16.0-108.stable CSI Addons 4.16.0-108.stable Succeeded openshift-storage odf-operator.v4.16.0-108.stable OpenShift Data Foundation 4.16.0-108.stable Succeeded openshift-storage odf-prometheus-operator.v4.16.0-108.stable Prometheus Operator 4.16.0-108.stable Succeeded openshift-storage recipe.v4.16.0-108.stable Recipe 4.16.0-108.stable Succeeded openshift-storage rook-ceph-operator.v4.16.0-108.stable Rook-Ceph 4.16.0-108.stable Succeeded 5.Create NodeNetworkConfigurationPolicy for each worker node apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: ceph-public-net-shim-argo005 namespace: openshift-storage spec: nodeSelector: node-role.kubernetes.io/worker: "" kubernetes.io/hostname: argo005.ceph.redhat.com desiredState: interfaces: - name: odf-pub-shim description: Shim interface used to connect host to OpenShift Data Foundation public Multus network type: mac-vlan state: up mac-vlan: base-iface: enp1s0f1 mode: bridge promiscuous: true ipv4: enabled: true dhcp: false address: - ip: 192.168.252.1 # STATIC IP FOR compute-0 prefix-length: 24 routes: config: - destination: 192.168.20.0/24 next-hop-interface: odf-pub-shim --- apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: ceph-public-net-shim-argo006 namespace: openshift-storage spec: nodeSelector: node-role.kubernetes.io/worker: "" kubernetes.io/hostname: argo006.ceph.redhat.com desiredState: interfaces: - name: odf-pub-shim description: Shim interface used to connect host to OpenShift Data Foundation public Multus network type: mac-vlan state: up mac-vlan: base-iface: enp1s0f1 mode: bridge promiscuous: true ipv4: enabled: true dhcp: false address: - ip: 192.168.252.2 # STATIC IP FOR compute-0 prefix-length: 24 routes: config: - destination: 192.168.20.0/24 next-hop-interface: odf-pub-shim --- apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: ceph-public-net-shim-argo007 namespace: openshift-storage spec: nodeSelector: node-role.kubernetes.io/worker: "" kubernetes.io/hostname: argo007.ceph.redhat.com desiredState: interfaces: - name: odf-pub-shim description: Shim interface used to connect host to OpenShift Data Foundation public Multus network type: mac-vlan state: up mac-vlan: base-iface: enp1s0f1 mode: bridge promiscuous: true ipv4: enabled: true dhcp: false address: - ip: 192.168.252.3 # STATIC IP FOR compute-0 prefix-length: 24 routes: config: - destination: 192.168.20.0/24 next-hop-interface: odf-pub-shim $ oc create -f NodeNetworkConfigurationPolicy.yaml nodenetworkconfigurationpolicy.nmstate.io/ceph-public-net-shim-argo005 created nodenetworkconfigurationpolicy.nmstate.io/ceph-public-net-shim-argo006 created nodenetworkconfigurationpolicy.nmstate.io/ceph-public-net-shim-argo007 created 6.Create multus-public-net.yaml with route apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: public-net namespace: openshift-storage labels: {} annotations: {} spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.20.0/24", "routes": [{"dst": "192.168.252.0/24"}] } }' 7.Create cluster-net apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: cluster-net namespace: openshift-storage labels: {} annotations: {} spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.30.0/24" } }' 8.Ruuning multus validation tool: $ ./rook multus validation run --public-network public-net --cluster-network cluster-net --namespace openshift-storage RESULT: multus validation test succeeded! 9.Create Storagecluster with multus $ oc describe pod noobaa-db-pg-0 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m37s default-scheduler 0/6 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling. Normal Scheduled 2m35s default-scheduler Successfully assigned openshift-storage/noobaa-db-pg-0 to argo005.ceph.redhat.com Normal SuccessfulAttachVolume 2m34s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-073ded78-220c-4fc7-98e0-09795fd22f64" Warning FailedMount 28s kubelet MountVolume.MountDevice failed for volume "pvc-073ded78-220c-4fc7-98e0-09795fd22f64" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Warning FailedMount 12s (x5 over 27s) kubelet MountVolume.MountDevice failed for volume "pvc-073ded78-220c-4fc7-98e0-09795fd22f64" : rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000001-b5778368-fed6-4740-85ce-ac5f72a45248 already exists
Deployment with test pass OCP Version:4.16.0-0.nightly-2024-05-23-173505 ODF Version: odf-operator.v4.16.0-113.stable Platform: BM Acceptance suite pass: https://url.corp.redhat.com/47cd3b0 I have a mistake in NAD configuration https://bugzilla.redhat.com/show_bug.cgi?id=2282543#c9 Changed master param from br-ex to enp1s0f1 Need to merge this PR for running multus in OCS-CI https://github.com/red-hat-storage/ocs-ci/pull/9777
Please update the RDT flag/text appropriately.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:4591