Bug 1845666

Description of problem (please be detailed as possible and provide log
snippests):
The scenario is simple.
an application pod is using our PVC (RBD/RWO). if the node the application pod is running on is going down (shutdown), it takes time until k8s understand the node is down (I think 20s), and then while the pod is moving to another location (node), the application pod hangs and waiting for k8s to release the lock on the PVC. IIRC k8s will never release the lock and wants a manual user intervention.
This of course basically fails any SLA the customer might have for the application (think PostgreSQL pod on a failed node moving to another node and never completing the startup process since it can't acquire lock on the PVC the previous pod used).

This bz is that we can track this issue from the application perspective.
bz 1795372 relates to the solution (I guess).
Version of all relevant components (if applicable):
any OCS4 version

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
the expect behavior is the application pod will migrate to the new node and that the PVC will move to it - thus not blocking any IOs on the application, at least not for an endless period of time.

Is there any workaround available to the best of your knowledge?
force kill the old application pod.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
yes

Can this issue reproduce from the UI?
yes

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. create a pod that needs a PVC from OCS of type block/RBD
2. kill or shutdown the *node* the pod is currently running
3. monitor the creation of the new pod.


Actual results:


Expected results:


Additional info:
From competitive perspective, Portworx have a solution for this (Annette checked it), when you fail a node, the application pod migrates to a new node and in matter of seconds gets the PVC attached to it.