Description of problem:
3 master and 6 works (m5.4xlarge)
6 workers: 2 in us-east-2a, 2 in us-east-2b, 2 in us-east-2c

75 Statefulsets behaves differently when they are in ONE project and when they are in 75 projects.

How does a gp2 PV choose its availability zone when it gets created?
What is the difference between one project and 75 projects?

Version-Release number of selected component (if applicable):
# oc get clusterversion version -o json | jq -r .status.desired
{
  "image": "registry.svc.ci.openshift.org/ocp/release@sha256:ef5a60a10812f2fa1e4c93a5042c1520ca55675f9b4085b08579510d71031047",
  "version": "4.0.0-0.nightly-2019-01-25-205123"
}

How reproducible: 3/3

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:



75 project, each has one statefulset with 2 and each pod has a PVC gernated by `volumeClaimTemplates`
# oc get pv --no-headers | wc -l
150
# oc get pod --all-namespaces | grep clusterb | grep Running | wc -l
150
# oc describe pv | grep zone  | grep 2a | wc -l
50
# oc describe pv | grep zone  | grep 2b | wc -l
50
# oc describe pv | grep zone  | grep 2c | wc -l
50


However, if 1 project has all 75 statefulsets.
# oc describe pv | grep zone  | grep 2a | wc -l
50
# oc describe pv | grep zone  | grep 2b | wc -l
0
# oc describe pv | grep zone  | grep 2c | wc -l
75

# oc describe pod -n clusteraproject37                            web0-0
Name:               web0-0
Namespace:          clusteraproject37
Priority:           0
PriorityClassName:  <none>
Node:               <none>
Labels:             app=server0
                    controller-revision-hash=web0-7c994b8d69
                    statefulset.kubernetes.io/pod-name=web0-0
Annotations:        openshift.io/scc=restricted
Status:             Pending
IP:                 
Controlled By:      StatefulSet/web0
Containers:
  server:
    Image:      openshift/hello-openshift
    Port:       8080/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     1
      memory:  256Mi
    Requests:
      cpu:        500m
      memory:     128Mi
    Environment:  <none>
    Mounts:
      /mydata from www (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z6hvb (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  www:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  www-web0-0
    ReadOnly:   false
  default-token-z6hvb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-z6hvb
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  14m (x25 over 14m)  default-scheduler  pod has unbound PersistentVolumeClaims (repeated 6 times)
  Warning  FailedScheduling  4m (x356 over 12m)  default-scheduler  0/9 nodes are available: 2 node(s) had no available volume zone, 3 node(s) had taints that the pod didn't tolerate, 4 node(s) exceed max volume count.