https://github.com/openshift/origin/pull/21409
Hmm, small modifications. 1. It is better to create a deployment rather than a plain pod for verifying this bug. Because if pod is backed by a deployment it will be migrated when node where it was running is shutdown. 2. You will notice that newly created pod (on new node) is stuck in ContainerCreating state while old pod (on shutdown node) is in Unknwown state. At this point you must force delete the old pod. Typically this can be done by: oc delete pod <pod> --force=true --grace-period=0 3. This should allow new pod to be running on new node.
Thanks HK! I have verified this bug on below version: openshift v3.10.79 kubernetes v1.10.0+b81c8f8 1. Create sc, pvc and dc # oc get dc NAME REVISION DESIRED CURRENT TRIGGERED BY vsphere 1 1 1 config # oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE azpvc Bound pvc-6696150c-f3af-11e8-8288-0050569f4627 1Gi RWO standard 18h # oc get pods NAME READY STATUS RESTARTS AGE vsphere-1-4kwj7 1/1 Running 0 2m # oc get pods vsphere-1-4kwj7 -o yaml | grep node nodeName: ocp310.node1.vsphere.local 2. Power off the node and delete the pod # oc get nodes NAME STATUS ROLES AGE VERSION ocp310.master.vsphere.local Ready master 49d v1.10.0+b81c8f8 ocp310.node1.vsphere.local NotReady compute 49d v1.10.0+b81c8f8 # oc delete pod vsphere-1-4kwj7 # oc get pods NAME READY STATUS RESTARTS AGE vsphere-1-4kwj7 1/1 Terminating 0 5m vsphere-1-vh4zl 0/1 ContainerCreating 0 10s 3. Then force delete the pod # oc delete pod vsphere-1-4kwj7 --force=true --grace-period=0 # oc get pods NAME READY STATUS RESTARTS AGE vsphere-1-vh4zl 0/1 ContainerCreating 0 4m # oc get pods NAME READY STATUS RESTARTS AGE vsphere-1-vh4zl 1/1 Running 0 11m # oc describe pods vsphere-1-vh4zl Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 11m default-scheduler Successfully assigned vsphere-1-vh4zl to ocp310.master.vsphere.local Warning FailedAttachVolume 11m attachdetach-controller Multi-Attach error for volume "pvc-6696150c-f3af-11e8-8288-0050569f4627" Volume is already used by pod(s) vsphere-1-4kwj7 Warning FailedMount 2m (x4 over 9m) kubelet, ocp310.master.vsphere.local Unable to mount volumes for pod "vsphere-1-vh4zl_default(ed1be1a9-f448-11e8-8288-0050569f4627)": timeout expired waiting for volumes to attach or mount for pod "default"/"vsphere-1-vh4zl". list of unmounted volumes=[azure]. list of unattached volumes=[azure default-token-t9wrs] Normal SuccessfulAttachVolume 1m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-6696150c-f3af-11e8-8288-0050569f4627" Normal Pulled 1m kubelet, ocp310.master.vsphere.local Container image "aosqe/hello-openshift" already present on machine Normal Created 1m kubelet, ocp310.master.vsphere.local Created container Normal Started 1m kubelet, ocp310.master.vsphere.local Started container This fix is awesome, even the node is shutdown we can still allow the disk to detach and attach to new node again. Thanks!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3750