Description of problem:

- It appears that aws-ebs-csi-driver-controller and aws-ebs-csi-driver-operator could get stuck at pending state if default cluster-wide node selector is set. 


Version-Release number of selected component (if applicable):

- 4.6.1


How reproducible:

- Always.


Steps to Reproduce:

1. Cluster-wide default node selector set:

$ oc get schedulers.config.openshift.io cluster -o jsonpath='{.spec.defaultNodeSelector}{"\n"}'
node-role.kubernetes.io/worker=

2. Pending:

$ oc get pod -n openshift-cluster-csi-drivers
NAME                                             READY   STATUS    RESTARTS   AGE
aws-ebs-csi-driver-controller-6cbb586dfc-pnzsh   0/6     Pending   0          2d17h
aws-ebs-csi-driver-node-6ffjj                    3/3     Running   0          2d17h
aws-ebs-csi-driver-node-79st2                    3/3     Running   0          2d17h
aws-ebs-csi-driver-node-fx7bv                    3/3     Running   0          2d17h
aws-ebs-csi-driver-node-g6dgz                    3/3     Running   0          2d17h
aws-ebs-csi-driver-node-jlx7f                    3/3     Running   0          2d17h
aws-ebs-csi-driver-node-kq2x5                    3/3     Running   0          2d17h
aws-ebs-csi-driver-node-lr5b4                    3/3     Running   0          2d17h
aws-ebs-csi-driver-node-lrxlf                    3/3     Running   0          2d17h
aws-ebs-csi-driver-node-r9jpq                    3/3     Running   0          2d17h
aws-ebs-csi-driver-node-t84j7                    3/3     Running   0          2d17h
aws-ebs-csi-driver-node-tppdw                    3/3     Running   0          2d17h
aws-ebs-csi-driver-node-xpnwp                    3/3     Running   0          2d17h
aws-ebs-csi-driver-operator-579f547595-4znd5     0/1     Pending   0          2d17h

3. Didn't match node selector:

$ oc get events --sort-by=.metadata.creationTimestamp -n openshift-cluster-csi-drivers
LAST SEEN   TYPE      REASON              OBJECT                                               MESSAGE
12h         Warning   FailedScheduling    pod/aws-ebs-csi-driver-operator-579f547595-4znd5     0/12 nodes are available: 12 node(s) didn't match node selector.
12h         Warning   FailedScheduling    pod/aws-ebs-csi-driver-controller-6cbb586dfc-pnzsh   0/12 nodes are available: 12 node(s) didn't match node selector.
.
.

4. openshift.io/node-selector:"" is not set:

$ oc get ns openshift-cluster-csi-drivers -o jsonpath='{.metadata.annotations}{"\n"}'
{"openshift.io/sa.scc.mcs":"s0:c18,c17","openshift.io/sa.scc.supplemental-groups":"1000340000/10000","openshift.io/sa.scc.uid-range":"1000340000/10000"}


Fix:

1. Annotate:

$ oc annotate ns/openshift-cluster-csi-drivers openshift.io/node-selector=""

2. Delete/restart the pods, and then they are running:

$ oc get pod
NAME                                             READY   STATUS    RESTARTS   AGE
aws-ebs-csi-driver-controller-6cbb586dfc-f55tc   6/6     Running   0          42m
aws-ebs-csi-driver-node-6ffjj                    3/3     Running   0          2d18h
aws-ebs-csi-driver-node-79st2                    3/3     Running   0          2d18h
aws-ebs-csi-driver-node-fx7bv                    3/3     Running   0          2d18h
aws-ebs-csi-driver-node-g6dgz                    3/3     Running   0          2d18h
aws-ebs-csi-driver-node-jlx7f                    3/3     Running   0          2d18h
aws-ebs-csi-driver-node-kq2x5                    3/3     Running   0          2d18h
aws-ebs-csi-driver-node-lr5b4                    3/3     Running   0          2d18h
aws-ebs-csi-driver-node-lrxlf                    3/3     Running   0          2d18h
aws-ebs-csi-driver-node-r9jpq                    3/3     Running   0          2d18h
aws-ebs-csi-driver-node-t84j7                    3/3     Running   0          2d18h
aws-ebs-csi-driver-node-tppdw                    3/3     Running   0          2d18h
aws-ebs-csi-driver-node-xpnwp                    3/3     Running   0          2d18h
aws-ebs-csi-driver-operator-579f547595-js86p     1/1     Running   0          41m


Conclusion:

- It appears we need openshift.io/node-selector="" on openshift-cluster-csi-drivers?