Description of problem:

This may be the root cause of bug 1912521, both have pods stuck in Terminating.

In a system test reliability cluster that periodically scales application deployments up/down from 1 replica->2->back to 1, some pods are getting stuck in Terminating on scale down and never clean up.

I have a must-gather and node journal showing this for two pods in our current cluster:

pod django-psql-persistent-1-bpngc in namespace django-psql-persistent-2
pod rails-pgsql-persistent-2-xkt9p in namespace rails-pgsql-persistent-19

Both pods were running on node ip-10-0-162-121.us-west-2.compute.internal

pod django-psql-persistent-1-bpngc is the more recent to get stuck.  It's scale down where it wedged was at 2021-01-05 05:55:21 UTC.   The first errors I see for it after that are:

Jan 05 05:55:22 ip-10-0-162-121 crio[1609]: time="2021-01-05 05:55:22.863160622Z" level=info msg="RunSandbox: releasing container name: k8s_POD_django-psql-persistent-1-bpngc_django-psql-persistent-2_03626057-68f2-4cac-9080-8c02dbc42666_0" id=4907b207-70b0-4f48-959c-e3603824c707 n>
Jan 05 05:55:22 ip-10-0-162-121 crio[1609]: time="2021-01-05 05:55:22.863223642Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_django-psql-persistent-1-bpngc_django-psql-persistent-2_03626057-68f2-4cac-9080-8c02dbc42666_0" id=4907b207-70b0-4f48-959c-e3603824c707 nam>
Jan 05 05:55:22 ip-10-0-162-121 hyperkube[1643]: E0105 05:55:22.864591    1643 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = error reading container (probably exited) json message: EOF
Jan 05 05:55:22 ip-10-0-162-121 hyperkube[1643]: E0105 05:55:22.864652    1643 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "django-psql-persistent-1-bpngc_django-psql-persistent-2(03626057-68f2-4cac-9080-8c02dbc42666)" failed: rpc error: code = Unknown desc = error reading>
Jan 05 05:55:22 ip-10-0-162-121 hyperkube[1643]: E0105 05:55:22.864671    1643 kuberuntime_manager.go:755] createPodSandbox for pod "django-psql-persistent-1-bpngc_django-psql-persistent-2(03626057-68f2-4cac-9080-8c02dbc42666)" failed: rpc error: code = Unknown desc = error readin>
Jan 05 05:55:22 ip-10-0-162-121 hyperkube[1643]: E0105 05:55:22.864724    1643 pod_workers.go:191] Error syncing pod 03626057-68f2-4cac-9080-8c02dbc42666 ("django-psql-persistent-1-bpngc_django-psql-persistent-2(03626057-68f2-4cac-9080-8c02dbc42666)"), skipping: failed to "CreateP>



Version-Release number of selected component (if applicable): 4.7.0-0.nightly-2021-01-04-153716


How reproducible: If it is the same as bug 1912521 then it is reliably reproducible.  I'll start a new cluster for this particular bug and try it again with just scale up/down testing.


Steps to Reproduce:
1.  OOTB AWS 3 master/3 worker cluster
2.  Run a workload that periodically creates/deletes projects, deployments and scales the deployments up/down.  


Actual results:

After about 12 hours, started seeing pods get stuck in terminating on scale down.  They never clean up



Additional info:

I will add full must-gather location and the journal of a node where the two pods described above were running before becoming wedged.