Bug 2030403

1. Proposed title of this feature request
Deploying LSO backed ODF on top of unschedulable control plane nodes (masters)

3. What is the nature and description of the request?
Being able to deploy LSO backed ODF on top of master nodes without making master nodes schedulable

4. Why does the customer need this? (List the business requirements here)
Especially for bare metal setup whith preset HW (customers have often contracts where a certain HW type is preferred/discounted,..) 
it often happens that also control plane nodes have more than sufficient power to run additional load to not wast that resources.
An other use case might be the setup is a to customer sold solution where the worker nodes should be as good as possible free from infra 
workload to be able to fully sell that compute power.
Making masters schedulable brings other problems with it, like the need to change the default scheduler or add additional labels
to custom namespaces

5. How would the customer like to achieve this? (List the functional requirements here)
Taint/label all relevant components for BM setup so ODF can be deployed on top of masters that have *not* been made schedulable

6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.
Label and taint all relevant object accordingly and verify those are  only running on the masters (like done in https://github.com/dmoessne/ocp-odf-iscsi-infra/tree/main/ocs/taint#validate-components-running-on-correct-nodes)

7. Is there already an existing RFE upstream or in Red Hat Bugzilla?

At least similar thin is in https://bugzilla.redhat.com/show_bug.cgi?id=1915607
And we already have https://access.redhat.com/articles/6408481 which would perhaps need just some polishing and addition of LSO.


8. Does the customer have any specific timeline dependencies and which release would they like to target (i.e. RHEL8, RHEL9)?
As soon as possible would be great and for ODF 4.8

9. Is the sales team involved in this request and do they have any additional input?
not at this point, but can be if needed

10. List any affected packages or components.
- LSO
- ODF


additional info 
PoC procedure in https://github.com/dmoessne/ocp-odf-iscsi-infra/tree/main/ocs/taint
but there RGW is currently missing. Also, I did a comparison between a deployment on masters and workers with special attention 
to affinities and only difference I found was for mds 
~~~

spec:                                                           spec:
  affinity:                                                       affinity:
    nodeAffinity:                                                   nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:                 requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:                                              nodeSelectorTerms:
        - matchExpressions:                                             - matchExpressions:
          - key: cluster.ocs.openshift.io/openshift-storage               - key: cluster.ocs.openshift.io/openshift-storage
            operator: Exists                                                operator: Exists
          - key: node-role.kubernetes.io/master               |     podAntiAffinity:
            operator: Exists                                  |       requiredDuringSchedulingIgnoredDuringExecution:
                                                              >       - labelSelector:
                                                              >           matchExpressions:
                                                              >           - key: app
                                                              >             operator: In
                                                              >             values:
                                                              >             - rook-ceph-mds
                                                              >         topologyKey: topology.kubernetes.io/zone
  containers:                                                     containers:

~~~

which is a bit confusing as my understanding is that we just overwrite settings that are there and hence I would have expected
the podAntiAffinity is stil there....

Also, it might be that topologyKey on zone makes less sense for a BM deployment