Description of problem:
When Prometheus pods are restarted the replay of the WAL consumes significantly more memory than steady state. A cluster that is currently healthy may deadlock on restart due to the increased consumption during boot up after a rolling restart or upgrade.


Version-Release number of selected component (if applicable):
4.3.2 but likely all versions

How reproducible:
100% when steady state consumption nears node resource limits

Steps to Reproduce:
1. Populate Prometheus data until it nears node resource limits
2. Restart Prometheus

Actual results:
Burst memory consumption causes Prometheus to crashloop during bootup with no prior warning that the system has exceeded healthy limits prior to the restart.

Expected results:
Either reduce the startup burst utilization or devise an alerting pattern that will ensure admins are aware they've exceeded the limits of available resources prior to critical events like power outage, rolling restarts, or upgrades.

Additional info:
This burst of consumption has knock on effects due to a kubelet bug which results in system processes being OOMKilled and further degradation of the node https://bugzilla.redhat.com/show_bug.cgi?id=1808429