Bug 2249678

Description of problem (please be detailed as possible and provide log
snippests):

The multus network address detection job does not derive placement from the CephCluster's "all" placement, only from "osd". This is a bug reported upstream here: https://github.com/rook/rook/issues/13138


This is also in the process of being fixed upstream here: https://github.com/rook/rook/pull/13206



Version of all relevant components (if applicable): ODF v4.14.0


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

No, but it might be an upgrade issue for some existing multus supportex customers. 


Is there any workaround available to the best of your knowledge?

A valid workaround is to have a user experiencing issues who is using the 'all' placement to manually specify cephcluster.spec.network.addressRanges for cluster/public networks. This will cause rook to skip its network address autodetection process.



Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

3 - somewhat complex since this requires multus AND CephCluster 'all' placement configs


Can this issue reproducible?

Yes. 


Can this issue reproduce from the UI?

Not sure


If this is a regression, please provide more details to justify this:

I believe this is a regression. Customers who are currently using Multus and the 'all' placement spec might hit this issue. Not all users will hit the issue; that depends on if the spec allows the detection job to run on another node in the cluster that has the requisite host networks.


Steps to Reproduce:

Taint all nodes in the openshift cluster, and then only set the toleration for said taint to the "all" section of the CephCluster.

For example, Use this taint...

kubectl taint nodes --all node-role.kubernetes.io/storage=true:NoSchedule


And this placement spec on CephCluster...

  placement:
    all:
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/storage
          operator: Equal
          value: "true"


Actual results:

rook-ceph-network-*-canary jobs will remain in pending with an error event like below:

Warning  FailedScheduling  12s   default-scheduler  0/3 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/storage: true}. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..



Expected results:

rook-ceph-network-*-canary jobs should be schedulable with 'all' placement settings.