Bug 1732209

Description of problem:
Could not dynamic create volume for ebs pvc.
We could dynamic create ebs volume on this server at the first, but failed to dynamic create volumes later.

Version-Release number of selected component (if applicable):
version   4.1.7    

How reproducible:
Not always.

Steps to Reproduce:
1.Create a pvc
oc describe pvc ebsc
Name:          ebsc
Namespace:     chaoyang
StorageClass:  gp2
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-class: gp2
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Events:
  Type       Reason                Age                    From                         Message
  ----       ------                ----                   ----                         -------
  Normal     WaitForFirstConsumer  76s (x903 over 3h41m)  persistentvolume-controller  waiting for first consumer to be created before binding
Mounted By:  mypod

2.Create pod
 oc describe pods mypod
Name:               mypod
Namespace:          chaoyang
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-0-69-82.us-east-2.compute.internal/10.0.69.82
Start Time:         Mon, 22 Jul 2019 17:49:51 +0800
Labels:             name=frontendhttp
Annotations:        openshift.io/scc: anyuid
Status:             Pending
IP:                 
Containers:
  myfrontend:
    Container ID:   
    Image:          jhou/hello-openshift
    Image ID:       
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /tmp from aws (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-8v6wl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  aws:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  ebs
    ReadOnly:   false
  default-token-8v6wl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-8v6wl
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                    From                                               Message
  ----     ------       ----                   ----                                               -------
  Warning  FailedMount  4m18s (x436 over 16h)  kubelet, ip-10-0-69-82.us-east-2.compute.internal  Unable to mount volumes for pod "mypod_chaoyang(0c414fd2-ac66-11e9-a974-0224b1289da4)": timeout expired waiting for volumes to attach or mount for pod "chaoyang"/"mypod". list of unmounted volumes=[aws]. list of unattached volumes=[aws default-token-8v6wl]


Actual results:
Pod is in status of 'ContainerCreating'

Expected results:
Pod is running

Master Log:
openshift-kube-controller-manager/pods/kube-controller-manager-ip-10-0-74-79.us-east-2.compute.internal/kube-controller-manager-5/kube-controller-manager-5/logs/current.log:2019-07-22T07:46:53.182387077Z I0722 07:46:53.182369       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"chaoyang", Name:"ebsc", UID:"9b1194e0-ac4f-11e9-a916-0224b1289da4", APIVersion:"v1", ResourceVersion:"112329", FieldPath:""}): type: 'Normal' reason: 'WaitForFirstConsumer' waiting for first consumer to be created before binding

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info: