Description of problem (please be detailed as possible and provide log
snippests):

On IBM cloud ( ROKS ) if we have 6 OCP nodes  and all of them labeled to serve as ODF nodes too

# oc get nodes
NAME           STATUS   ROLES           AGE   VERSION
10.240.0.23    Ready    master,worker   14d   v1.21.4+6438632
10.240.0.24    Ready    master,worker   14d   v1.21.4+6438632
10.240.128.6   Ready    master,worker   14d   v1.21.4+6438632
10.240.128.7   Ready    master,worker   14d   v1.21.4+6438632
10.240.64.10   Ready    master,worker   14d   v1.21.4+6438632
10.240.64.9    Ready    master,worker   14d   v1.21.4+6438632

[root@perf148b ~]# oc get nodes --show-labels |grep storage |awk '{print $1}'
10.240.0.23
10.240.0.24
10.240.128.6
10.240.128.7
10.240.64.10
10.240.64.9

Then if creaed ODF cluster on this setup the following will happen

-> adding first three OSDs will ensure that we get 1 OSD per node
-> adding additional three will lead that these will land on same nodes as first 3 OSDs.
To ensure that every node gets one OSD, we have to "cordon" node, delete one OSD and then it will start on new node where 
no OSD runs. 

-> if we add 3 more OSDs ( 6->9 OSDs ) they will spread one per node
-> if we add 3 more OSDs ( 9-> OSDs ) we will have situation that some nodes will get 3 OSDs while other will stay on 1 OSD - eg nodes "10.240.64.10" - 3 OSD, and node "10.240.64.9" - 1 OSD in below output 


# oc get pods  -o wide|grep osd |grep -v Com |grep -v drop
rook-ceph-osd-0-5fd67b877-bhj5z                                   2/2     Running     0          26h    172.17.113.166   10.240.128.6   <none>           <none>
rook-ceph-osd-1-76b4648f55-hnxsd                                  2/2     Running     0          26h    172.17.83.157    10.240.0.23    <none>           <none>
rook-ceph-osd-10-77786c9b7d-459bf                                 2/2     Running     0          24m    172.17.83.185    10.240.0.23    <none>           <none>
rook-ceph-osd-11-68cdb6b48b-6svzv                                 2/2     Running     0          23m    172.17.99.176    10.240.64.10   <none>           <none>
rook-ceph-osd-2-784ff6bb85-hk47l                                  2/2     Running     3          26h    172.17.88.69     10.240.64.9    <none>           <none>
rook-ceph-osd-3-696956c7c9-gz7f7                                  2/2     Running     0          26h    172.17.116.5     10.240.128.7   <none>           <none>
rook-ceph-osd-4-75c65759bb-rfl78                                  2/2     Running     0          26h    172.17.99.132    10.240.64.10   <none>           <none>
rook-ceph-osd-5-5557f5b757-qlrnm                                  2/2     Running     0          26h    172.17.112.217   10.240.0.24    <none>           <none>
rook-ceph-osd-6-85675f89fd-tj6bn                                  2/2     Running     0          42m    172.17.116.18    10.240.128.7   <none>           <none>
rook-ceph-osd-7-7dc69c6f6d-5vgdt                                  2/2     Running     0          42m    172.17.99.160    10.240.64.10   <none>           <none>
rook-ceph-osd-8-674486cbf5-7xbg7                                  2/2     Running     0          42m    172.17.112.211   10.240.0.24    <none>           <none>
rook-ceph-osd-9-76d976f4df-f8kb8                                  2/2     Running     0          24m    172.17.113.164   10.240.128.6   <none>           <none>


From Performance  / Scale Perspective this is not ideal. 


Version of all relevant components (if applicable):

OCP v4.8 / ODF v4.8 Installed on IBM cloud 

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
No

Is there any workaround available to the best of your knowledge?
Yes

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
3

Can this issue reproducible?
yes

Can this issue reproduce from the UI?
NA

Steps to Reproduce:
1. Install 6 nodes OCP / ODF cluster and start adding OSDs to cluster 


Actual results:
OSD pods will not be spread equally across ODF nodes 


Expected results:
Every node to get same number of OSDs 


Additional info:

We expand cluster with :

# oc edit storagecluster -n openshift-storage 
increase count for storageDeviceSets: 

storageDeviceSets:
  - config: {}
    count: 4
    dataPVCTemplate:
      metadata: {}


Attached will be storagecluster.yml and must-gather from affected cluster.