Created attachment 1485428 [details]
prometheus container logs

Description of problem:
prometheus container throws out error
"write data/wal/000001: file already closed"  recently, a few days ago it did not have this issue.
$ oc -n openshift-devops-monitor logs -c prometheus prometheus-0 
level=warn ts=2018-09-21T05:06:32.477030856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed"
level=warn ts=2018-09-21T05:06:32.478906866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed"
level=warn ts=2018-09-21T05:06:32.479216896Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed"
level=warn ts=2018-09-21T05:06:32.593700506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed"
level=warn ts=2018-09-21T05:06:32.594301784Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed"
level=warn ts=2018-09-21T05:06:32.905714099Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed"
level=warn ts=2018-09-21T05:06:33.215831934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed"


Version-Release number of selected component (if applicable):

OpenShift Master:
    v3.11.7 
Kubernetes Master:
    v1.11.0+d4cacc0 
OpenShift Web Console:
    v3.11.7 

How reproducible:
recently

Steps to Reproduce:
1. oc -n openshift-devops-monitor logs -c prometheus prometheus-0 
2.
3.

Actual results:
prometheus container throws out error
"write data/wal/000001: file already closed" 

Expected results:
Should not have error

Additional info: