Bug 2232464

Description of problem (please be detailed as possible and provide log
snippests):
- CSI pods have priority 0 instead of using openshift priorityclasses (https://docs.openshift.com/container-platform/4.13/nodes/pods/nodes-pods-priority.html).  Customer workloads also have priority 0 by default, which leads to a race for scheduling and resources.  

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
- Customer have upcoming activities which this issue may occur again 


Is there any workaround available to the best of your knowledge?
1. The most practical way is by moving a pod that has at least 350 of MB of requests to a different node.

2. If the 1st does not help then contact us (Red Hat) with the node name which we can recycle the whole node with help from SRE.


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)? 3


Can this issue reproducible? Yes
This was triggered via a DR testing scenario.  So I think trying to quickly spin up full workloads on an empty cluster

Can this issue reproduce from the UI?
- Not sure


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Apply CSI pods and customer workloads at the same time
2. Repeat until race is hit where CSI pods cannot be scheduled.


Actual results:
CSI pods can't be scheduled when customer workloads get there first

Expected results:
CSI pods are scheduled before customer workloads

Additional info:
Per SRE - From OCP point of view, AWS EBS CSI driver on ROSA should have enough priority - linking 4.11 yamls:

https://github.com/openshift/aws-ebs-csi-driver-operator/blob/release-4.11/assets/node.yaml#L24

https://github.com/openshift/aws-ebs-csi-driver-operator/blob/release-4.11/assets/controller.yaml#L25